- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper investigates why increasing test-time compute for self-evaluation does not consistently close the gap between final-answer accuracy and pass@N. The authors measure pairwise misranking between correct and incorrect candidates and decompose expected ranking error into squared bias and variance. Empirically, repeated evaluation generally reduces variance, while squared bias initially decreases before approaching a plateau.
The paper introduces SCORPIO, which repeatedly partitions candidates into random groups, jointly scores each group, and averages each candidate’s scores across repetitions. Experiments use Qwen3.5-4B and GPT-OSS-20B on mathematics, coding, and reasoning-puzzle benchmarks, with an additional Gemini-3.1-Pro experiment on Terminal-Bench-2.1. The authors also apply SCORPIO to partial-solution pruning and recursive aggregation. Reported improvements include up to seven percentage points in verification accuracy and 6–42% lower cost for matching or exceeding the strongest baseline’s peak accuracy, alongside improvements in search and aggregation.
Critical strengths
- A simple, training-free procedure delivers practical improvements across multiple settings. SCORPIO requires only changes to candidate grouping and scoring, making it relatively easy to integrate into existing inference pipelines. Figure 2 demonstrates favorable accuracy–cost tradeoffs across two models and three task categories. Including generation and evaluation expenses in the reported costs strengthens the practical relevance of these results.
- The ablations provide useful evidence about algorithmic design choices. The K=2 comparison in Figure 13 helps distinguish the effect of random pairing from that of increasing group size. Figures 4 and 15 further show that larger groups introduce tradeoffs. These experiments substantiate the useful observation that more elaborate pairing strategies do not necessarily use evaluation compute more effectively in the studied settings.
Critical weaknesses
-
The analysis does not sufficiently connect the measured decomposition to the final-answer selection bottleneck. For each candidate pair, the decomposition is pᵢⱼ=pᵢⱼ²+pᵢⱼ(1-pᵢⱼ), so both components are determined by the same misranking probability. This describes whether errors occur consistently, but does not establish how those errors determine top-1 selection accuracy. Average error over all correct–incorrect pairs and correctness at the top of the ranking capture different properties: rankings can have identical decomposition terms yet produce opposite selection outcomes.
Equation (1) also counts every tie as an error. With discrete scores, persistent ties can therefore contribute to the observed pattern of declining variance and residual squared bias. The analysis should distinguish strict misrankings from failures to differentiate candidates and quantify how these components relate to actual selection failures. The results support a plateau within the evaluated budgets, but the proposed explanation of that plateau remains incomplete.
-
The core scoring procedure substantially overlaps with prior work, while the extension experiments do not sufficiently isolate the additional contribution. SCORPIO’s central sequence—shuffle candidates, partition them into groups, jointly produce individual scores, repeat, and average—closely matches the Shuffled-then-Batched self-consistency procedure in Korikov et al. Section 3 of the prior paper describes these operations. Appendix B acknowledges this connection, but the stated distinctions primarily concern the task, analysis, and applications; the algorithmic novelty remains unclear.
Consequently, the search and aggregation extensions are important to establishing the paper’s contribution. However, the search experiments change both intermediate pruning and final-answer selection, leaving their respective contributions unresolved. The aggregation analysis highlights contamination by unsupported tool calls, but lacks a simple output-format filtering baseline. These omissions make it difficult to assess how much of the extension gains specifically requires SCORPIO’s scoring design.
Questions for authors
- Contribution clarification: With the scoring function, group size, and repetition count held fixed, what substantive differences remain between Algorithm 1 and Korikov et al.’s STB procedure? If the core algorithms coincide, what should be regarded as this paper’s principal new contribution?
- Analysis and estimation clarification: What are the correct–incorrect tie rates across methods and budgets, and how are ties resolved during final selection? Furthermore, the plug-in estimator in Appendix C.1 satisfies
which overestimates squared bias and underestimates variance at finite R. Do the main plateau observations and differences between methods remain after correcting this estimation bias and separating ties from strict misrankings? 3. Targeted validation needed for attribution: When all search methods use SCORPIO for final selection and total costs are matched, how much improvement remains attributable to intermediate pruning? For RSA, how much additional benefit does SCORPIO provide after introducing a simple filter for invalid outputs or unsupported tool calls? These controls would materially affect my assessment of the extension contributions. 4. Cost-accounting clarification: Appendix D.2 charges ThinkPRM-1.5B the same per-token prices as the generating model. What justifies this assumption? Do the relevant accuracy–cost comparisons remain favorable under model-appropriate cost estimates or measured execution costs?
7 comments
Author Response - Part 2/2
[.. continued from part 1]
W2/Q3. With the same final selector, Scorpio's search gives the largest gain from search.
We add an experiment on HMMT with GPT-OSS-20B where all search methods use Scorpio to select their final answer (Fig. 27, App G.7). Starting from the pass@1 of independent sampling (57.5%), we report the gain from search (pass@1 of the candidates remaining after search) and the further gain from Scorpio's final selection. DeepConf spends no compute on evaluating partial solutions, so we run it with larger to keep total costs comparable.
| Method | Gain from search (pp) | Gain from selection (pp) | Top-1 (%) | |
|---|---|---|---|---|
| DeepConf | 16 | +1.9 | +17.6 | 77.0 |
| 64 | +2.1 | +25.1 | 84.7 | |
| 256 | +2.8 | +24.8 | 85.1 | |
| ThinkPRM-1.5B | 8 | +3.4 | +14.8 | 75.7 |
| 32 | +1.2 | +24.4 | 83.1 | |
| 128 | +1.9 | +24.7 | 84.1 | |
| Scorpio | 8 | +5.4 | +13.3 | 76.2 |
| 32 | +4.4 | +21.1 | 83.0 | |
| 128 | +4.8 | +23.3 | 85.6 |
Scorpio's search yields the largest gain (4.4–5.4 pp vs. 1.9–2.8 for DeepConf and 1.2–3.4 for ThinkPRM), so it prunes poor partial solutions most accurately. For all methods, final selection contributes more than search. After selection, Scorpio and DeepConf scale comparably, and Scorpio reaches ThinkPRM's accuracy at about half the cost.
Pruning intermediate solutions using Scorpio search saves compute: compared with sampling solutions independently and selecting with Scorpio, Scorpio's search reaches comparable accuracy at about 4× lower cost and improves peak accuracy by 1.2 pp (Fig. 28, App G.7).
W2/Q3. Scorpio's gains in aggregation go beyond simple output-format filtering.
We add a baseline that filters out responses containing tool calls after each aggregation loop, so that aggregation sets are built only from the remaining responses. The table reports the peak pass@1 (%) over aggregation loops on LCBv6 with GPT-OSS-20B at :
| RSA | RSA + Filter | RSA + Scorpio |
|---|---|---|
| 64.7 | 68.1 | 74.7 |
Filtering improves RSA's peak pass@1 by 3.4 pp, and Scorpio improves over the filtered baseline by a further 6.6 pp. Unlike RSA + Scorpio, RSA + Filter still degrades as the population size grows (Fig. 21, App G.2).
Q4. With model-appropriate pricing, Scorpio still outperforms ThinkPRM.
We now price ThinkPRM-1.5B at 0.18 USD per million input and output tokens, based on www.pricepertoken.com. With the revised pricing, Scorpio's peak top-1 accuracy is 2.4 points higher than ThinkPRM's on AIME 2024 and 10.7 points higher on LCBv6 (Fig. 26, App G.6).
We hope that the rebuttal clarifies questions raised by the reviewer. We would be happy to discuss any further questions about the work, and would really appreciate an increase in score if reviewers' concerns are adequately addressed to facilitate acceptance of the paper.
Gentle Reminder
Dear Reviewer 8uKz,
As the discussion period is nearing its end, we wanted to kindly check whether our response has addressed your concerns. We would be happy to clarify anything pending concerns!
Author Response - Part 1/2
We thank the reviewer for their detailed feedback. We are glad that the reviewer found the proposed method simple and training-free, with practical accuracy-cost improvements across test-time algorithms. Below, we address the questions raised by the reviewer with references to App G, where we add the rebuttal experiments. Changes to existing parts of the paper are marked in red.
W2/Q1. Our primary contribution is the bias–variance lens on test-time compute. The scoring procedure is an outcome of this analysis. We will state the overlap with STB in the main text.
We agree that with the scoring function, group size and number of repetitions factored out, Scorpio's scoring procedure coincides with STB, and we will state this explicitly in the main text. However, the two works operate in different settings: Korikov et al. use a judge to rank retrieved passages based on relevance to a query, while we study whether a model can rank its own generations.
We examine test-time compute through the lens of the bias–variance decomposition, which explains how the design choices of a scorer affect the test-time scaling of an algorithm:
In short, while our scoring procedure is similar to STB, our analysis explains when additional test-time compute stops helping and which design choices move that limit. A key finding of this analysis is that simply grouping candidates at random can match or outperform tournament-based approaches, which have become a popular way to scale test-time compute for verification.
W1. Pairwise ranking error closely tracks top-1 error.
Across 2 models × 3 datasets, top-1 error (1 − top-1 accuracy) and pairwise ranking error decrease together as inference compute increases (Fig. 24, App G.4). The table reports their Pearson correlation:
We agree that two rankings can share the same decomposition terms yet differ at the top, which is why our main results report both metrics. Empirically, we find that the two metrics move together, as the table shows. We provide additional motivation for choosing pairwise ranking error in Common Response P2.
W1. At high compute, ranking error is dominated by bias, consistent with the plateau in top-1 accuracy.
We run Scorpio with GPT-OSS-20B for twice as many scoring rounds (M = 10 → 20). On three datasets, variance falls by 25–27%, while bias changes by at most 4%. Fitting a curve to the variance shows that it extrapolates to near zero, while a non-zero bias remains. Together with the strong correlation above, this residual bias is consistent with the plateau in top-1 accuracy. Further details are provided in App G.5.
W1/Q2. With unbiased estimators and random tie-breaking, the bias plateau and the differences between methods remain.
We have updated App C.1 to (i) break ties uniformly at random, and (ii) use unbiased estimators of squared bias and variance. Fig. 23 (App G.3) recomputes the decomposition for GPT-OSS-20B with the revised definitions: variance still decreases with compute while bias plateaus, and Scorpio still attains similar or lower ranking error than pointwise scoring and V1 at matched compute.
Ties are common only at low compute, so the revised definitions mainly lower the measured bias there. The table reports the fraction of correct–incorrect pairs with tied scores at the lowest and highest compute budgets:
At the highest budgets, fewer than 5% of pairs are tied, so the remaining ranking error comes mostly from strict misrankings and the residual bias is not an artifact of ties.
Q2. Ties at final selection are broken randomly for pointwise scoring and Scorpio. For V1, we follow the authors' implementation: candidates are ordered by win rate, with ties broken by average score.