acceptodds
Under review as a conference paper at ICLR 2027

On the Limits of Test-Time Compute for Self-Evaluation

Abstract

Self-verification and aggregation have emerged as popular approaches for bridging the gap between and . However, increasing the number of candidates does not consistently narrow the gap between final-answer accuracy and . To understand this limitation, we use ranking error to measure how reliably models distinguish their correct generations from incorrect ones. We show that ranking error admits a bias-variance decomposition, with both components shaped by algorithmic design choices and inference compute. Across our experiments, additional compute generally reduces variance, while bias initially decreases before approaching a plateau. Guided by this analysis, we introduce Scorpio, a simple and general method that jointly scores randomly grouped candidates. Across math, code, and puzzle benchmarks, Scorpio improves verification, tree search, and aggregation. For verification, Scorpio improves accuracy over the strongest baseline by up to percentage points while matching or exceeding that baseline’s peak accuracy with – lower total inference cost. For tree search, using Scorpio to score partial solutions and select the final answer improves accuracy by up to 10 percentage points over prior methods. Finally, Scorpio improves the scaling of aggregation-based methods by percentage points on coding tasks.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Test-time scaling, self-evaluation, LLM-as-a-judge, reasoning, inference efficiency, self-verification

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_8uKzOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper investigates why increasing test-time compute for self-evaluation does not consistently close the gap between final-answer accuracy and pass@N. The authors measure pairwise misranking between correct and incorrect candidates and decompose expected ranking error into squared bias and variance. Empirically, repeated evaluation generally reduces variance, while squared bias initially decreases before approaching a plateau.

The paper introduces SCORPIO, which repeatedly partitions candidates into random groups, jointly scores each group, and averages each candidate’s scores across repetitions. Experiments use Qwen3.5-4B and GPT-OSS-20B on mathematics, coding, and reasoning-puzzle benchmarks, with an additional Gemini-3.1-Pro experiment on Terminal-Bench-2.1. The authors also apply SCORPIO to partial-solution pruning and recursive aggregation. Reported improvements include up to seven percentage points in verification accuracy and 6–42% lower cost for matching or exceeding the strongest baseline’s peak accuracy, alongside improvements in search and aggregation.

Critical strengths
  1. A simple, training-free procedure delivers practical improvements across multiple settings. SCORPIO requires only changes to candidate grouping and scoring, making it relatively easy to integrate into existing inference pipelines. Figure 2 demonstrates favorable accuracy–cost tradeoffs across two models and three task categories. Including generation and evaluation expenses in the reported costs strengthens the practical relevance of these results.
  2. The ablations provide useful evidence about algorithmic design choices. The K=2 comparison in Figure 13 helps distinguish the effect of random pairing from that of increasing group size. Figures 4 and 15 further show that larger groups introduce tradeoffs. These experiments substantiate the useful observation that more elaborate pairing strategies do not necessarily use evaluation compute more effectively in the studied settings.
Critical weaknesses
  1. The analysis does not sufficiently connect the measured decomposition to the final-answer selection bottleneck. For each candidate pair, the decomposition is pᵢⱼ=pᵢⱼ²+pᵢⱼ(1-pᵢⱼ), so both components are determined by the same misranking probability. This describes whether errors occur consistently, but does not establish how those errors determine top-1 selection accuracy. Average error over all correct–incorrect pairs and correctness at the top of the ranking capture different properties: rankings can have identical decomposition terms yet produce opposite selection outcomes.

    Equation (1) also counts every tie as an error. With discrete scores, persistent ties can therefore contribute to the observed pattern of declining variance and residual squared bias. The analysis should distinguish strict misrankings from failures to differentiate candidates and quantify how these components relate to actual selection failures. The results support a plateau within the evaluated budgets, but the proposed explanation of that plateau remains incomplete.

  2. The core scoring procedure substantially overlaps with prior work, while the extension experiments do not sufficiently isolate the additional contribution. SCORPIO’s central sequence—shuffle candidates, partition them into groups, jointly produce individual scores, repeat, and average—closely matches the Shuffled-then-Batched self-consistency procedure in Korikov et al. Section 3 of the prior paper describes these operations. Appendix B acknowledges this connection, but the stated distinctions primarily concern the task, analysis, and applications; the algorithmic novelty remains unclear.

    Consequently, the search and aggregation extensions are important to establishing the paper’s contribution. However, the search experiments change both intermediate pruning and final-answer selection, leaving their respective contributions unresolved. The aggregation analysis highlights contamination by unsupported tool calls, but lacks a simple output-format filtering baseline. These omissions make it difficult to assess how much of the extension gains specifically requires SCORPIO’s scoring design.

Questions for authors
  1. Contribution clarification: With the scoring function, group size, and repetition count held fixed, what substantive differences remain between Algorithm 1 and Korikov et al.’s STB procedure? If the core algorithms coincide, what should be regarded as this paper’s principal new contribution?
  2. Analysis and estimation clarification: What are the correct–incorrect tie rates across methods and budgets, and how are ties resolved during final selection? Furthermore, the plug-in estimator in Appendix C.1 satisfies
E[p^ 2]=p2+p(1−p)R,\mathbb{E}[\hat p^{\,2}] =p^2+\frac{p(1-p)}{R},

which overestimates squared bias and underestimates variance at finite R. Do the main plateau observations and differences between methods remain after correcting this estimation bias and separating ties from strict misrankings? 3. Targeted validation needed for attribution: When all search methods use SCORPIO for final selection and total costs are matched, how much improvement remains attributable to intermediate pruning? For RSA, how much additional benefit does SCORPIO provide after introducing a simple filter for invalid outputs or unsupported tool calls? These controls would materially affect my assessment of the extension contributions. 4. Cost-accounting clarification: Appendix D.2 charges ThinkPRM-1.5B the same per-token prices as the generating model. What justifies this assumption? Do the relevant accuracy–cost comparisons remain favorable under model-appropriate cost estimates or measured execution costs?

7 comments
AuthorsOpenReview ↗

Author Response - Part 1/2

We thank the reviewer for their detailed feedback. We are glad that the reviewer found the proposed method simple and training-free, with practical accuracy-cost improvements across test-time algorithms. Below, we address the questions raised by the reviewer with references to App G, where we add the rebuttal experiments. Changes to existing parts of the paper are marked in red.

W2/Q1. Our primary contribution is the bias–variance lens on test-time compute. The scoring procedure is an outcome of this analysis. We will state the overlap with STB in the main text.

We agree that with the scoring function, group size and number of repetitions factored out, Scorpio's scoring procedure coincides with STB, and we will state this explicitly in the main text. However, the two works operate in different settings: Korikov et al. use a judge to rank retrieved passages based on relevance to a query, while we study whether a model can rank its own generations.

We examine test-time compute through the lens of the bias–variance decomposition, which explains how the design choices of a scorer affect the test-time scaling of an algorithm:

  • Group size: more candidates in a scorer's context lower variance but raise bias. Grouping four candidates achieves a good trade-off.
  • Grouping strategy: placing similar candidates in the context can help lower bias, but increases variance.
  • Scoring rounds: additional rounds drive variance toward zero while bias plateaus.
  • Beyond verification: the same scorer improves search, by pruning poor partial solutions, and aggregation, by selecting good candidates.

In short, while our scoring procedure is similar to STB, our analysis explains when additional test-time compute stops helping and which design choices move that limit. A key finding of this analysis is that simply grouping candidates at random can match or outperform tournament-based approaches, which have become a popular way to scale test-time compute for verification.

W1. Pairwise ranking error closely tracks top-1 error.

Across 2 models × 3 datasets, top-1 error (1 − top-1 accuracy) and pairwise ranking error decrease together as inference compute increases (Fig. 24, App G.4). The table reports their Pearson correlation:

HMMTRG GamesLCBv6
GPT-OSS-20B0.990.961.00
Qwen3.5-4B0.990.990.99

We agree that two rankings can share the same decomposition terms yet differ at the top, which is why our main results report both metrics. Empirically, we find that the two metrics move together, as the table shows. We provide additional motivation for choosing pairwise ranking error in Common Response P2.

W1. At high compute, ranking error is dominated by bias, consistent with the plateau in top-1 accuracy.

We run Scorpio with GPT-OSS-20B for twice as many scoring rounds (MM = 10 → 20). On three datasets, variance falls by 25–27%, while bias changes by at most 4%. Fitting a curve to the variance shows that it extrapolates to near zero, while a non-zero bias remains. Together with the strong correlation above, this residual bias is consistent with the plateau in top-1 accuracy. Further details are provided in App G.5.

W1/Q2. With unbiased estimators and random tie-breaking, the bias plateau and the differences between methods remain.

We have updated App C.1 to (i) break ties uniformly at random, and (ii) use unbiased estimators of squared bias and variance. Fig. 23 (App G.3) recomputes the decomposition for GPT-OSS-20B with the revised definitions: variance still decreases with compute while bias plateaus, and Scorpio still attains similar or lower ranking error than pointwise scoring and V1 at matched compute.

Ties are common only at low compute, so the revised definitions mainly lower the measured bias there. The table reports the fraction of correct–incorrect pairs with tied scores at the lowest and highest compute budgets:

Tied pairs (%)HMMTLCBv6
Pointwise21.7→2.717.2→4.2
V110.8→0.89.8→0.1
Scorpio15.9→0.721.7→1.3

At the highest budgets, fewer than 5% of pairs are tied, so the remaining ranking error comes mostly from strict misrankings and the residual bias is not an artifact of ties.

Q2. Ties at final selection are broken randomly for pointwise scoring and Scorpio. For V1, we follow the authors' implementation: candidates are ordered by win rate, with ties broken by average score.

AuthorsOpenReview ↗

Author Response - Part 2/2

[.. continued from part 1]

W2/Q3. With the same final selector, Scorpio's search gives the largest gain from search.

We add an experiment on HMMT with GPT-OSS-20B where all search methods use Scorpio to select their final answer (Fig. 27, App G.7). Starting from the pass@1 of independent sampling (57.5%), we report the gain from search (pass@1 of the candidates remaining after search) and the further gain from Scorpio's final selection. DeepConf spends no compute on evaluating partial solutions, so we run it with larger NN to keep total costs comparable.

MethodNNGain from search (pp)Gain from selection (pp)Top-1 (%)
DeepConf16+1.9+17.677.0
64+2.1+25.184.7
256+2.8+24.885.1
ThinkPRM-1.5B8+3.4+14.875.7
32+1.2+24.483.1
128+1.9+24.784.1
Scorpio8+5.4+13.376.2
32+4.4+21.183.0
128+4.8+23.385.6

Scorpio's search yields the largest gain (4.4–5.4 pp vs. 1.9–2.8 for DeepConf and 1.2–3.4 for ThinkPRM), so it prunes poor partial solutions most accurately. For all methods, final selection contributes more than search. After selection, Scorpio and DeepConf scale comparably, and Scorpio reaches ThinkPRM's accuracy at about half the cost.

Pruning intermediate solutions using Scorpio search saves compute: compared with sampling N=128N=128 solutions independently and selecting with Scorpio, Scorpio's search reaches comparable accuracy at about 4× lower cost and improves peak accuracy by 1.2 pp (Fig. 28, App G.7).

W2/Q3. Scorpio's gains in aggregation go beyond simple output-format filtering.

We add a baseline that filters out responses containing tool calls after each aggregation loop, so that aggregation sets are built only from the remaining responses. The table reports the peak pass@1 (%) over aggregation loops on LCBv6 with GPT-OSS-20B at N=128N=128:

RSARSA + FilterRSA + Scorpio
64.768.174.7

Filtering improves RSA's peak pass@1 by 3.4 pp, and Scorpio improves over the filtered baseline by a further 6.6 pp. Unlike RSA + Scorpio, RSA + Filter still degrades as the population size NN grows (Fig. 21, App G.2).

Q4. With model-appropriate pricing, Scorpio still outperforms ThinkPRM.

We now price ThinkPRM-1.5B at 0.18 USD per million input and output tokens, based on www.pricepertoken.com. With the revised pricing, Scorpio's peak top-1 accuracy is 2.4 points higher than ThinkPRM's on AIME 2024 and 10.7 points higher on LCBv6 (Fig. 26, App G.6).


We hope that the rebuttal clarifies questions raised by the reviewer. We would be happy to discuss any further questions about the work, and would really appreciate an increase in score if reviewers' concerns are adequately addressed to facilitate acceptance of the paper.

AuthorsOpenReview ↗

Gentle Reminder

Dear Reviewer 8uKz,

As the discussion period is nearing its end, we wanted to kindly check whether our response has addressed your concerns. We would be happy to clarify anything pending concerns!

Reviewer_GhggOpenReview ↗
Rating
3
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies why additional test-time compute does not reliably close the gap between selection accuracy and pass@N. It analyzes pairwise ranking error through a bias–variance decomposition and introduces SCORPIO, which repeatedly scores randomly grouped candidates and averages their ratings. Experiments are conducted on verification, search and aggregation using Qwen3.5-4B and GPT-OSS-20B on math, coding and reasoning benchmarks, with an additional Gemini-3.1-Pro experiment on Terminal-Bench 2.1.

Critical strengths
  • The pass@1-pass@N gap not shrinking with more candidates is an important finding, and the bias-variance view helps separate inconsistent judgments from persistent misrankings. The analysis of group size and pairing strategy in Figs. 4 and 13 makes this concrete.
  • SCORPIO is simple and applies to verification, search and aggregation. It improves peak verification accuracy on math and reasoning tasks, and reaches similar coding accuracy at lower cost.
  • The experiments use repeated runs with confidence intervals, and the cost accounting includes generation as well as evaluation. The example in Appendix C.2 nicely shows how joint comparison catches errors that pointwise scoring misses.
Critical weaknesses
  • The ranking analysis is not well connected to the selection objective. Since both bias and variance are functions of the same misranking probability, bias plateauing while variance vanishes is largely expected from the definition (Appendix C.3). Ties also count as full errors, which may penalize pointwise scoring that often gives tied ratings. The authors should report results on how well ranking error predicts top-1 accuracy and with random tie-breaking
  • The search comparisons do not isolate groupwise scoring. The methods differ in both intermediate evaluation and final selection, ThinkPRM is adapted to a different search procedure, and its 1.5B evaluator is priced at the generating model’s rates. Pointwise and pairwise scoring under the same search procedure should be compared.
  • The aggregation gains are not well explained. In Fig. 7, the tool-call example suggests that much of the gain may come from filtering format violations, which a rule-based check could also catch. RSA with a validity filter should be compared, and aggregation is only evaluated with GPT-OSS-20B on LCBv6.
  • The frontier-scale evidence is limited. The Terminal-Bench experiment in Fig. 5 has no pointwise or V1 baseline, and it scores candidate summaries, so the remaining gap may come from missing evidence rather than the judge itself.
Questions for authors
  • How much of the residual bias comes from self-preference? Would a different or stronger judge evaluating the same candidates lower the bias floor, rather than only the variance?
  • In Appendix C.2, the error is caught when the group contains different final answers. Would grouping candidates across distinct answer clusters reduce bias further?
5 comments
AuthorsOpenReview ↗

Author Response - Part 1/2

We thank the reviewer for their detailed feedback, and are glad that the reviewer found the persistent pass@1–pass@N gap an important finding, the bias–variance view useful, and Scorpio simple and effective across multiple test-time algorithms. We also appreciate the reviewer noting our experimental rigour, with confidence intervals and unified cost accounting across experiments. Below, we address the concerns raised by the reviewer with references to App G, where we add the rebuttal experiments.

W1. With random tie-breaking, Scorpio still matches or outperforms pointwise scoring and V1. Ties are common only at low compute, so the correction lowers bias for all methods there.

Following the reviewer's suggestion, we now break ties uniformly at random, and also use unbiased estimators of squared bias and variance (App C.1). Fig. 23 recomputes the decomposition for GPT-OSS-20B with the revised definitions, and the trends continue to hold. The table below reports the fraction of pairs with tied scores at the lowest and highest compute budgets:

Tied pairs (%)HMMTLCBv6
Pointwise21.7→2.717.2→4.2
V110.8→0.89.8→0.1
Scorpio15.9→0.721.7→1.3

At the highest budgets, fewer than 5% of pairs are tied, so pointwise scoring is not disadvantaged by the tie convention at matched compute.

W1. Bias plateauing while variance vanishes is not implied by the definition; it reflects consistent misrankings by the judge.

Although bias and variance are functions of the same probability pp, the definition does not enforce that bias plateaus and variance decreases as inference compute increases. To illustrate this, consider a judge that assigns both the correct and incorrect candidate a score of 0 or 10 with equal probability. In this setting, the probability of misranking this pair is p=0.5p=0.5, and its variance p(1−p)p(1-p) stays at 0.25 regardless of the compute spent. Whether variance vanishes or not therefore depends on how often the judge guesses.

In App G.5, we show that this guessing scenario is infrequent: running Scorpio with double the verification compute, variance approaches zero while bias remains constant. This is only possible if, for some fraction of pairs the misranking probability approaches 1, i.e., some pairs are consistently misranked.

Further, the definition does not determine how fast variance decreases or where bias plateaus. Both depend on design choices of the judge. Our experiments show that these choices affect the scaling behavior of test-time algorithms. For instance, V1's slower variance reduction from pairing similar candidates (Fig. 13) is consistent with its slower improvement in top-1 accuracy with inference compute (Fig. 2).

W1. Ranking error strongly predicts top-1 error.

Across 2 models × 3 datasets, top-1 error (1 − top-1 accuracy) and pairwise ranking error decrease together as inference compute increases (Fig. 24, App G.4). The table reports their Pearson correlation:

HMMTRG GamesLCBv6
GPT-OSS-20B0.990.961.00
Qwen3.5-4B0.990.990.99

The plateau in bias and the vanishing variance, together with this strong correlation are consistent with the plateau in top-1 accuracy reported in the paper.

In Common Response P2, we explain our rationale for using pairwise ranking error to measure ranking quality.

W2. With the same final selector, Scorpio's search gives the largest gain from search.

We add an experiment on HMMT with GPT-OSS-20B where all search methods use Scorpio to select their final answer (Fig. 27, App G.7). Starting from the pass@1 of independent sampling (57.5%), we report the gain from search (pass@1 of the candidates remaining after search) and the further gain from Scorpio's final selection. DeepConf spends no compute on evaluating partial solutions, so we run it with larger NN to keep total costs comparable.

MethodNNGain from search (pp)Gain from selection (pp)Top-1 (%)
DeepConf16+1.9+17.677.0
64+2.1+25.184.7
256+2.8+24.885.1
ThinkPRM-1.5B8+3.4+14.875.7
32+1.2+24.483.1
128+1.9+24.784.1
Scorpio8+5.4+13.376.2
32+4.4+21.183.0
128+4.8+23.385.6

Scorpio's search yields the largest gain (4.4–5.4 pp vs. 1.9–2.8 for DeepConf and 1.2–3.4 for ThinkPRM), so it prunes poor partial solutions most accurately. For all methods, final selection contributes more than search. After selection, Scorpio and DeepConf scale comparably, and Scorpio reaches ThinkPRM's accuracy at about half the cost.

Pruning intermediate solutions using Scorpio search saves compute: compared with sampling N=128N=128 solutions independently and selecting with Scorpio, Scorpio's search reaches comparable accuracy at about 4× lower cost and improves peak accuracy by 1.2 pp (Fig. 28, App G.7).

AuthorsOpenReview ↗

Author Response - Part 2/2

[.. continued from part 1]

W2. With model-appropriate pricing, Scorpio still outperforms ThinkPRM.

We now price ThinkPRM-1.5B at 0.18 USD per million input and output tokens, based on www.pricepertoken.com. With the revised pricing, Scorpio's peak top-1 accuracy is 2.4 points higher than ThinkPRM's on AIME 2024 and 10.7 points higher on LCBv6 (Fig. 26, App G.6).

W2. Our search procedure does not disadvantage ThinkPRM: it outperforms ThinkPRM's original beam search at lower cost.

Results in the main text use the same search procedure for ThinkPRM and Scorpio. This procedure is adapted from DeepConf: it prunes unpromising partial solutions, so after every evaluation fewer solutions remain than at the start. In contrast, ThinkPRM's implementation uses beam search, which generates as many complete solutions as it starts with.

To check that this choice does not hurt ThinkPRM, we ran ThinkPRM using the authors' implementation. On AIME 2024 with GPT-OSS-20B (N=8N=8, 3 seeds), our search procedure improves over ThinkPRM's original search by 2.5 pp, while being about 3.6× cheaper. The cost savings come from two sources. (i) Although both start with the same number of solutions, our procedure prunes solutions along the way, which saves generation compute. (ii) We call ThinkPRM once every 2K tokens, whereas the original implementation calls it at every double newline (\n\n), up to 20 times per solution.

W3. Scorpio's gains in aggregation go beyond filtering format violations, and hold on a second model and dataset.

We add a baseline that filters out responses containing tool calls after each aggregation loop. The table reports the peak pass@1 (%) over aggregation loops on LCBv6 with GPT-OSS-20B at N=128N=128:

RSARSA + FilterRSA + Scorpio
64.768.174.7

Filtering improves RSA's peak pass@1 by 3.4 pp, and Scorpio improves over the filtered baseline by a further 6.6 pp. Fig. 21 (App G.2) shows complete scaling trends for different values of NN.

We also evaluate aggregation with a different model and task: Qwen3-4B-Instruct-2507 on AIME 2024 with population size N=64N=64. The table compares RSA and RSA + Scorpio at matched cost, with RSA's cost as xx (Fig. 22, App G.2):

MethodCostPass@1 (%)
RSAxx79.3
RSA + Scorpio0.93$x$84.1

W4. Scorpio outperforms pointwise scoring, V1 [1], and PPT [2] on Terminal-Bench 2.1. It improves over the strongest baseline by 1.3–3.1 pp at matched compute (Fig. 20, App G.1).

W4. Scoring summaries follows prior work showing they are better suited for selection than full trajectories.

While summaries can leave out some information, [3] notes that "full trajectories are too long and noisy to serve as reliable objects of comparison" (Section 3.2.1 and Fig. 4).

Q1. A different judge lowers the bias floor, not just the variance.

Thanks for the suggestion! When responses generated by Qwen3.5-4B are judged by GPT-OSS-20B instead of by Qwen3.5-4B itself the ranking error drops mainly because the residual bias drops:

HMMT (math)

JudgeBiasVarianceRanking errorTop-1 (%)
Qwen3.5-4B (self)0.1270.0340.16168.5
GPT-OSS-20B0.0420.0260.06874.7

LCBv6 (code)

JudgeBiasVarianceRanking errorTop-1 (%)
Qwen3.5-4B (self)0.1690.0680.23655.6
GPT-OSS-20B0.0870.0430.12969.9

The lower ranking error is accompanied by higher top-1 selection accuracy (Fig. 29, App G.8).

Q2. Grouping by distinct answers does not reduce bias; random grouping does better.

On HMMT with GPT-OSS-20B, we bucket responses by their final answer and build groups from different buckets. The table below reports bias, variance, ranking error, and top-1 accuracy after 8 scoring rounds, and Fig. 30 (App G.9) shows how these metrics change with inference compute.

GroupingBiasVarianceRanking errorTop-1 (%)
Distinct answers0.0550.0530.10879.0
Random (Scorpio)0.0420.0340.07682.8

This suggests that diversity across scoring rounds may matter more than diversity within a single round. Grouping by distinct answers makes each round diverse, but across rounds it discourages grouping candidates that share the same answer. Random grouping, on the other hand, imposes no such constraint.


[1] Singh et al. V1: Unifying Generation and Self-Verification for Parallel Reasoners. ICML, 2026.

[2] Kwok et al. LLM-as-a-Verifier: A General-Purpose Verification Framework. arXiv:2607.05391, 2026.

[3] Kim et al. Scaling Test-Time Compute for Agentic Coding. arXiv:2604.16529, 2026.


We hope that the rebuttal addresses the concerns raised by the reviewer. In particular, we hope the cross-judge and distinct-answer experiments address the reviewer's questions about the bias floor. We would be happy to discuss any further questions, and would really appreciate an increase in score if the concerns are adequately addressed.

AuthorsOpenReview ↗

Gentle Reminder

Dear Reviewer Ghgg,

As the discussion period is nearing its end, we wanted to kindly check whether our response has addressed your concerns. We would be happy to clarify anything pending concerns!

Reviewer_hLQAOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper investigates why increasing test-time compute for self-evaluation does not consistently close the gap between final-answer accuracy and pass@N. It analyzes pairwise ranking error through a bias-variance decomposition, finding that repeated evaluation generally reduces variance while residual ranking bias persists within the evaluated budgets. The paper further introduces SCORPIO, which repeatedly scores candidates jointly in randomly formed groups and averages their ratings. Experiments evaluate SCORPIO for answer verification, partial-trajectory pruning, and recursive aggregation, showing improved accuracy-cost trade-offs over the evaluated baselines while leaving a substantial gap to oracle selection.

Critical strengths
  • SCORPIO improves verification accuracy and cost efficiency by repeatedly scoring randomly grouped candidates. The results make the method a useful, easily adopted alternative to more elaborate evaluation schedules.

  • The paper applies the same scoring strategy to partial-trajectory pruning and recursive aggregation, demonstrating benefits across multiple inference settings.

Critical weaknesses
  • For a pair with misranking probability p, the decomposition is p=p^2+p(1-p), so both components are determined by the same probability. The defined “bias” can decrease through averaging even when expected candidate scores remain unchanged, and therefore does not isolate systematic scoring errors. Moreover, average pairwise error is not directly aligned with top-1 selection, since one consistently overrated incorrect candidate can cause selection failure despite low overall ranking error. These issues weaken the paper’s central explanation of why additional evaluation compute fails to close the pass@N gap.

  • The downstream experiments do not sufficiently isolate what produces the gains. In search, SCORPIO changes both intermediate pruning and final-answer selection, while baselines use different final selectors. The reported improvements therefore do not establish the benefit of SCORPIO’s partial-trajectory evaluation. In aggregation, comparison against unfiltered RSA leaves unclear whether the gains require randomized groupwise scoring or would also follow from simpler filtering, particularly for the highlighted output-format failure.

Questions for authors
  • What evidence distinguishes the reported bias plateau from finite-budget saturation of repeated scoring? In particular, do persistently reversed expected score margins explain the remaining top-1 selection failures, and how closely does pairwise ranking error track those failures?

  • How much of SCORPIO’s improvement comes from intermediate pruning versus final-answer selection? Do the gains persist when pruning methods share the same final selector, and how does search compare with independent sampling followed by SCORPIO at matched total cost?

5 comments
AuthorsOpenReview ↗

Author Response - Part 1/1

We thank the reviewer for their insightful suggestions and for their positive assessment of Scorpio as a simple and effective method that improves multiple test-time scaling algorithms. Below, we address the concerns raised by the reviewer with references to App G.

W1/Q1. Doubling the scoring compute does not reduce bias, so the plateau reflects systematic misrankings rather than finite-budget saturation.

To check whether more inference compute reduces bias further, we double the number of scoring rounds MM from 10 to 20 with GPT-OSS-20B:

DatasetBias M=10→20M=10 \to 20Variance M=10→20M=10 \to 20
HMMT0.042 → 0.0420.031 → 0.022
RG Games0.079 → 0.0810.050 → 0.037
LCBv60.065 → 0.0670.046 → 0.033

Bias changes by at most 4%, while variance decreases by 25–27%, and fitting a curve shows that the asymptotic variance approaches 0 on all three datasets. A non-zero bias with vanishing variance is only possible when, for some correct–incorrect pairs, the misranking probability approaches 1, i.e., the judge consistently ranks the incorrect candidate above the correct one. Further details are in App G.5.

W1/Q1. Pairwise ranking error closely tracks top-1 error, and the persistent misrankings are consistent with the remaining top-1 failures.

We agree that pairwise ranking error and top-1 accuracy capture different properties, which is why our main results report both. In Fig. 24 we show how ranking error varies with top-1 error (1 - top-1 accuracy) as inference compute is increased. The table below reports the Pearson correlation across 2 models × 3 datasets:

HMMTRG GamesLCBv6
GPT-OSS-20B0.990.961.00
Qwen3.5-4B0.990.990.99

As shown above, variance approaches 0 at high compute, so the remaining ranking error comes mostly from persistently reversed pairs (bias). Because ranking error is strongly correlated with top-1 error, these persistent misrankings track the remaining top-1 failures.

W2. Scorpio's gains in aggregation go beyond simple filtering.

We add a baseline that filters out responses containing tool calls after each aggregation loop. The table reports the peak pass@1 over aggregation loops on LCBv6 with GPT-OSS-20B at N=128N=128:

RSARSA + FilterRSA + Scorpio
64.768.174.7

Filtering improves RSA's peak pass@1 by 3.4 pp, and Scorpio improves over the filtered baseline by a further 6.6 pp. Fig. 21 (App G.2) shows that, unlike RSA + Scorpio, RSA + Filter still degrades as the population size NN grows.

W2/Q2. With the same final selector, Scorpio's search gives the largest gain from search.

We add an experiment on HMMT with GPT-OSS-20B where all search methods use Scorpio to select their final answer (Fig. 27, App G.7). Starting from the pass@1 of independent sampling (57.5%), we report the gain from search (pass@1 of the candidates remaining after search) and the further gain from Scorpio's final selection. DeepConf spends no compute on evaluating partial solutions, so we run it with larger NN to keep total costs comparable.

MethodNNGain from search (pp)Gain from selection (pp)Top-1 (%)
DeepConf16+1.9+17.677.0
64+2.1+25.184.7
256+2.8+24.885.1
ThinkPRM-1.5B8+3.4+14.875.7
32+1.2+24.483.1
128+1.9+24.784.1
Scorpio8+5.4+13.376.2
32+4.4+21.183.0
128+4.8+23.385.6

Scorpio's search yields the largest gain (4.4–5.4 pp vs. 1.9–2.8 for DeepConf and 1.2–3.4 for ThinkPRM), so it prunes poor partial solutions most accurately. For all methods, final selection contributes more than search. After selection, Scorpio and DeepConf scale comparably, and Scorpio reaches ThinkPRM's accuracy at about half the cost.

Pruning intermediate solutions using Scorpio search saves compute: compared with sampling N=128N=128 solutions independently and selecting with Scorpio, Scorpio's search reaches comparable accuracy at about 4× lower cost and improves peak accuracy by 1.2 pp (Fig. 28, App G.7).


We hope that the rebuttal clarifies questions raised by the reviewer. We would be happy to discuss any further questions about the work, and would really appreciate an increase in score if reviewers' concerns are adequately addressed to facilitate acceptance of the paper.

AuthorsOpenReview ↗

Gentle Reminder

Dear Reviewer hLQA,

As the discussion period is nearing its end, we wanted to kindly check whether our response has addressed your concerns. We would be happy to clarify anything pending concerns!

Dear Reviewer hLQA,

Given that the rebuttal period ends in a few hours, we wanted to check whether our rebuttal addresses your concerns. We would be happy to discuss any further questions about the work. If these revisions resolve your remaining questions, we would greatly appreciate it if you could reconsider your score. We have added a brief summary of the changes we made during the rebuttal as a common message.

Comments on the paper

2 comments
AuthorsOpenReview ↗

Common Response

We thank the reviewers for their careful reading of our work. In this common response, we (P1) report additional baselines on agentic tasks, (P2) explain our choice of pairwise ranking error, and (P3) show that using a different model as a judge can reduce bias. New results are in App G, and changes to existing parts of the paper are marked in red.

P1. On Terminal-Bench 2.1 with Gemini-3.1-Pro, Scorpio outperforms pointwise scoring, V1, and PPT.

The main text reported results for only our method on Terminal-Bench 2.1. App G.1 provides three additional baselines: (i) pointwise scoring, (ii) V1 [1], and (iii) Probabilistic Pivot Tournament (PPT) [2]. The table reports the peak top-1 accuracy (%) of each method with N=8N=8 candidates:

PointwisePPTV1Scorpio
76.077.578.680.8

Scorpio improves over the strongest baseline, V1, by 1.3–3.1 percentage points at matched compute (Fig. 20). These results mirror our findings on math, puzzles, and code in the main text: randomly grouping candidates scales better than tournament-based approaches.

P2. Rationale for choosing pairwise ranking error as a measure of ranking quality

All reviewers raised concerns about the relation between pairwise ranking error and top-1 accuracy. In the individual responses, we note that the two metrics are highly correlated (Pearson r = 0.96–1.00 across 2 models × 3 datasets).

Our choice of pairwise ranking error was motivated by two observations:

  • Not all test-time algorithms select a single best candidate. Both tree search and aggregation benefit from a good set of candidates: tree search benefits from having good partial solutions in the beam, while aggregation works better when the model is conditioned on a good set of candidates. Therefore, we wanted a metric that looks beyond the top-1. Pairwise ranking error is one such metric, and it is commonly used in the ranking literature to measure ranking quality [4].
  • Even in verification, where a single best candidate is picked, several recent methods such as V1 [1], PPT [2], and RTV [3] rely on repeated pairwise comparisons of the form "Is solution 1 better than solution 2?". Pairwise ranking error directly measures the accuracy of such judgments.

P3. Using a different model as a judge lowers bias and improves top-1 accuracy.

Results in the main text showed that as inference compute increased, variance went down, but bias stopped decreasing. To mitigate this residual bias, we follow the suggestion from Reviewer Ghgg and use a different model as a judge. Solutions are generated by Qwen3.5-4B and evaluated either by the same model (self-judge) or by GPT-OSS-20B. As noted in App G.8, using GPT-OSS-20B reduces bias by 49–67% relative to self-judging and improves top-1 accuracy by 6.2–14.3 percentage points.


[1] Singh et al. V1: Unifying Generation and Self-Verification for Parallel Reasoners. ICML, 2026.

[2] Kwok et al. LLM-as-a-Verifier: A General-Purpose Verification Framework. arXiv:2607.05391, 2026.

[3] Kim et al. Scaling Test-Time Compute for Agentic Coding. arXiv:2604.16529, 2026.

[4] Cortes and Mohri. AUC Optimization vs. Error Rate Minimization. NeurIPS, 2003.

AuthorsOpenReview ↗

Rebuttal Summary

Below we summarize the strengths noted by the reviewers, and the most relevant concerns together with the actions we took to resolve them. New results are in App. G.

Strengths noted by the reviewers

  • hLQA: Scorpio is "a useful, easily adopted alternative to more elaborate evaluation schedules," "demonstrating benefits across multiple inference settings."
  • Ghgg: The pass@1–pass@N gap is "an important finding," and "the bias-variance view helps separate inconsistent judgments from persistent misrankings."
  • 8uKz: "A simple, training-free procedure delivers practical improvements across multiple settings," and "the ablations provide useful evidence about algorithmic design choices."

Main concerns and our actions

QueryReviewersOur rebuttal
Does pairwise ranking error reflect selection quality (top-1 acc)?hLQA, Ghgg, 8uKzPairwise ranking error is strongly correlated with top-1 acc, with a Pearson correlation of 0.96–1.00 across 2 models × 3 datasets (App. G.4).
Are gains in aggregation just output-format filtering?hLQA, Ghgg, 8uKzA newly added filtering baseline shows that our method does more than format filtering: Scorpio improves over it by 6.6 points (App. G.2).
Isolating the improvements from search vs. selection in search experimentshLQA, Ghgg, 8uKzWe report pass@1 of candidates before selection. Scorpio improves pass@1 over independent sampling by 4.4–5.4 pp, compared with 1.2–3.4 pp for the baselines (App. G.7). With the same final selector, Scorpio's search reaches ThinkPRM-1.5B's accuracy at about half the cost.
Limited baselines on agentic tasksGhggOn Terminal-Bench 2.1, newly added baselines (V1, PPT, and pointwise scoring) show that Scorpio outperforms them by 1.3–3.1 pp at matched compute (App. G.1).
Can the residual bias be reduced?GhggWe thank the reviewer for this suggestion. Using a different model as a judge reduces bias by 49–67% and raises top-1 accuracy by 6.2–14.3 pp (App. G.8). We plan to add this strong result to the main paper.

We once again thank the reviewers for the time spent on reviewing our work. We would be happy to address any concerns in the remaining time.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.