- Rating
- 3
- Confidence
- 2
- Informativeness
- 3
- Presentation
- 2
- Soundness
- 3
Read review
Paper summary
This paper asks whether oracle coverage identifies the best system under a deployed selector. It proves that strict oracle dominance can coexist with plurality regret approaching one, derives sharp accuracy bounds from partial categorical information, and examines ranking reversals empirically. A secondary contribution skips verifier queries while preserving the full weighted-vote decision.
Critical strengths
-
The sharp identification bounds are the strongest independent contribution. Proposition 2 characterizes the range of voting accuracies compatible with retained correct-category information, abstention mass, and the largest wrong-answer probability. This goes beyond evaluating or allocating samples under a specified distribution.
-
Proposition 1 gives a useful consequence for system selection: even complete oracle curves can support the wrong deployment choice. The empirical example is substantial—at , low effort has 2.86 percentage points higher oracle coverage but 7.64 points lower plurality accuracy than high effort.
-
The empirical checks are careful. The main reversal survives simultaneous correction and invalid-vote controls, while parser sensitivity and stable overall-winner cases are reported explicitly.
Critical weaknesses
-
The related work should discuss PETS (Liu et al., 2026). Its §2 and Appendix C.3 use answer distributions, population voting targets, and vote margins, sharing the foundations of this paper’s §2 and Appendix D. The novel contribution lies in cross-system identification and sharp bounds, rather than the underlying voting mechanism.
-
The practical use of the bounds remains less established than their theoretical value. The empirical ranges exclude distribution-estimation uncertainty, including unseen categories. The pilot also does not demonstrate an advantage over pass@1 or direct selector evaluation.
-
The deployment evidence has a limited scope. Including Qwen leaves the overall winner unchanged, effort settings have different generation costs, and the reported 51–70% time savings cover verification only. The query-skipping result is a useful application of decision preservation, but should remain secondary to the identification contribution.
Questions for authors
-
How sensitive are identification-based decisions to uncertainty in the largest wrong-answer mass and to unseen answer categories?
-
What practical setting benefits from partial-information bounds when direct selector evaluation is available?
-
What fraction of total pipeline time is spent in verification under the measured configuration?
5 comments
Response to Reviewer FbAA (Part 2/3)
Q2. Which practical setting benefits when direct selector evaluation is available?
Direct evaluation measures a selector at the budget it samples, and estimating plurality@ directly needs at least responses per question for every candidate. The bounds add three things.
- Accuracy at budgets that were not sampled. Evaluations often use fewer samples than deployment. On MATH-500, where accuracy keeps rising with , plurality@16 misses the 64-sample accuracy by 4.4 points, and its interval contains it in only 47% of draws. The bias-corrected range misses by 1.4 points, and its interval always contains it. Pass@1 misses the level by 3.7 to 28.0 points, and extrapolated pass@ overstates it by 11 to 26 points. The advantage of the range grows with the deployment budget (new Table 25). From the same 16 samples, it misses the 128- and 256-sample accuracies by 1.6 and 1.5 points, and its intervals contain them in 99% and 100% of draws. Plurality@16 misses them by 5.4 and 5.5 points, and its interval contains them in only 30% and 29% of draws. On MMLU-Pro, from 4 of 32 samples per question, the range misses the 32-sample accuracy by 0.4 points and its interval always contains it, while the interval around plurality@4 contains it in only 1% of draws.
- Certified comparisons. At ( on the new banks), orderings certified by the range intervals were correct in every draw on every bank. From 16 samples, those certified at and 256 were all correct as well. On the Scorio Trace pairs that pass@ orders the wrong way, extrapolated pass@ agrees with plurality in only 24 to 43% of draws, while the corrected range agrees in 68 to 75% (pass@1: 64 to 69%). On the new SuperGPQA bank, the intervals certify the plurality order on these pairs in 94% of draws.
- Screening. A system is dropped once another system is certified better than it. From 16 samples per question, this drops 3 of the 4 SuperGPQA systems, 6 of the 7 MATH-500 systems and 15 to 18 of the 20 Scorio Trace systems, and never the system that is best at . Even if every remaining system is then evaluated directly at , screening saves 56 to 64% of the generated samples (Section 4.2, new Table 22). On MMLU-Pro, 4 samples per question drop 5 of the 7 systems, never the best one, and save 62% at . On MATH-500 the saving grows with the deployment budget, to 75% at and 80% at . The ranges need only per-question answer frequencies, so they can also be computed when only those are reported.
The question-disjoint pilot (Section 4.2) shows what is at stake. Development questions choose the system and held-out questions test it, and choosing by oracle coverage instead of plurality loses 2.32 points on the held-out half (simultaneous 95% CI ). Compared with pass@1 and with direct evaluation at the smaller budget, the bounds give the accurate level at unsampled budgets, certified orderings and cheaper screening, and their advantage widens at larger deployment budgets (Table 2, Appendix E.2).
Response to Reviewer FbAA (Part 3/3)
W3. Scope of the deployment evidence.
- Overall winner. The census adds banks where the pass@ winner and the plurality winner differ. On MATH-500, choosing by pass@ at costs 17.6 to 18.8 points when invalid outputs count as votes. Among the seven new 3.8B to 8B SuperGPQA models, pass@32 picks Llama-3.1-8B, whose plurality accuracy is 6.3 points below Qwen2.5-7B's (simultaneous interval ; Section 4.1). This pair is itself a significant reversal, with the same signs in each half of the 3,600 questions. On 3,600 MMLU-Pro questions, pass@32 again picks Llama-3.1-8B, 9.8 points below Qwen2.5-7B (), and this pair is again a significant reversal.
- Generation cost of the effort settings. We repeated the effort comparison at equal generated tokens (new Appendix D.4, Table 15). A high-effort response uses on average 10,671 completion tokens and a low-effort one 355, so with the tokens of one high-effort sample, low effort draws 30. It then has 33.3 points more oracle coverage but 5.3 points lower plurality accuracy (simultaneous interval ). Of all 11 token-matched comparisons between the three settings, 9 are resolved reversals under both validity rules. Low-effort plurality accuracy stays between 39.7% and 40.3% from 30 to 80 samples, while high effort reaches 45.0% with one sample, so this ordering does not depend on how the cost of long responses is counted.
- Verification-only savings. The certificate acts on the verifier stage. Q3 below places that stage in the whole pipeline.
Q3. Fraction of pipeline time spent in verification. We measured it on one L40S, the GPU type used for all verifier timings (Section 4.4; Appendix C.4, new Table 8). The MATH-500 pools of Base and GRPO-42 were generated with vLLM on one L40S per shard, eight questions and 256 samples per call, and the time of every call is logged. Amortized over the 500 questions, generating 64 candidates costs 6.5 s per question for Base and 5.8 s for GRPO-42. Full PRM-7B scoring of the same 64 candidates takes 5.5 to 9.8 s per question, depending on how responses are grouped into blocks. Verification is therefore 46 to 63% of end-to-end time in this configuration, and the certificate, which cuts verifier time by 51 to 70%, shortens the whole pipeline by 30 to 36%.
The code and outputs of every new analysis are in the updated supplementary material and in the anonymous repository linked in the abstract.
Thank you again. Your comments on scope and practical use made the paper clearer about when the bounds help. If the new results resolve your concerns, we would be grateful if you would take them into account in your final assessment.
Response to Reviewer FbAA (Part 1/3)
Thank you for the precise review and for the clear description of what the paper contributes. As you suggested, the revision puts the identification results first, and the query-skipping certificate now appears in Section 4.4 as an application of decision preservation.
W1. PETS. Section 5 now discusses PETS (Liu et al., 2026) and the closely related Feng et al. (ICML 2026), and it makes explicit, as you put it, that our contribution lies in cross-system identification and the sharp bounds. PETS models each question's answers as a categorical distribution and defines the self-consistency rate as the probability that a finite majority vote agrees with the population majority. It allocates samples across questions with a Gaussian approximation to multinomial vote margins (its Appendix C.3). Feng et al. bound how fast self-consistency finds the mode through the margin (p1−p2)2. Both ask how a vote approaches one model's population mode under a given distribution. We ask three different questions:
The two views fit together: in a PETS-style allocation, a question whose range is already decided needs no further samples.
W2/Q1. Sensitivity to uncertainty in the largest wrong mass and to unseen categories.
Unseen categories do not enter the ranges. Proposition 2 already takes the worst and best case over every way the unobserved wrong mass could be spread, including over any number of answers that were never sampled. Only the strongest wrong answer matters. Lemma 1 shows that the error in its estimate does not grow with the number of distinct answers. The new Proposition 3 bounds the competing mass using only the count of the true strongest competitor, so the bound holds even when that answer never appears in the sample.
The experiments now include this estimation uncertainty (Section 4.2, Table 2, Appendix E). We re-estimate c and w from n saved responses per question and predict plurality accuracy at K=64. At n=16, the bias-corrected range is the most accurate rule on each bank, within 0.4, 1.6 and 1.4 points on SuperGPQA, Scorio Trace and MATH-500. A two-level bootstrap gives intervals that contain the 64-sample accuracy in every draw. Every ordering they certify is correct: on average 5.3 of 6 pairs on SuperGPQA, 122 of 190 on Scorio Trace, 15 of 21 on MATH-500, 51 of 55 on the new 11-system SuperGPQA bank from 8 of 32 responses, and 19 of 21 on MMLU-Pro. On the pairs that pass@K orders the wrong way, they never certify the pass@K order.