acceptodds
Under review as a conference paper at ICLR 2027

Scoring Signal Selection for Test-Time Scaling

Abstract

Test-time scaling (TTS) improves reasoning by sampling multiple candidate solutions and selecting among them with a scoring function. Scoring signals include internal signals (e.g., model confidence), which add negligible cost beyond generation, and external signals (e.g., process reward models, PRMs), which provide additional evaluation at extra cost. Prior work largely compares or combines such signals under a fixed scoring rule across instances. We ask whether the scoring signal can be selected per instance after candidate generation but before invoking external scoring. To our knowledge, this post-generation selection problem has not been explicitly studied. In this work, we systematically study post-generation scoring signal selection. We empirically show that internal and external signals exhibit low rank correlation when evaluating the same candidate sets, while each uniquely succeeds on a non-trivial subset of instances. This complementarity creates a meaningful opportunity for per-instance signal selection. We then evaluate practical selection strategies, ranging from simple candidate disagreement statistics to learned routing models. We find that inexpensive disagreement statistics such as answer entropy capture useful coarse routing signals, while the learned selectors we evaluate do not consistently improve upon them across the full accuracy–call-rate frontier, even with labeled supervision and richer representations. These findings establish scoring signal selection as a practically relevant and largely unexplored axis for improving TTS efficiency, while showing that fine-grained per-instance selection remains challenging beyond coarse disagreement signals.

Primary area:
other topics in machine learning (i.e., none of the above)
Keywords:
Test-Time Scaling, Process Reward Models, LLM

Reviews

Among:
Shape:
1234this paper, Bottom 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.25 over 4 reviews, above 49% of the others.
Reviewer_DZv3OpenReview ↗
Rating
3
Confidence
4
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper examines per-instance scoring signal selection for test-time scaling, a mechanism that decides whether to apply low-cost internal confidence or an external process reward model (PRM) after candidate generation. It verifies the complementary strengths of the two scoring signals and shows that simple disagreement-based heuristics, with answer entropy being the most representative, achieve competitive accuracy-efficiency trade-offs when compared against learned selectors.

Critical strengths
  • The paper identifies an interesting and practically relevant direction for improving test-time scaling efficiency, with insights supported by empirical evidence of complementarity between internal and external scoring signals.
  • The experiments compare different selection strategies and investigate their behavior across models, scoring signals, and sampling budgets. The finding that simple heuristics remain competitive with learned selectors is particularly interesting.
Critical weaknesses

The proposed scoring signal selection is only evaluated under Best-of-N, although Appendix A.2 shows that alternative aggregation methods, such as weighted majority voting, substantially reduce the complementarity between scoring signals. It remains unclear whether the proposed approach would provide similar benefits under these alternative aggregation strategies.

Questions for authors

The current framework invokes the PRM on all candidates whenever external scoring is selected. Could selectively evaluating only a subset of candidates further improve the accuracy–efficiency tradeoff?

4 comments

W1. Does routing remain useful beyond BoN?

We agree that the aggregation rule affects the benefit of scoring-signal selection. As shown in Appendix A.2, switching from BoN to WMV reduces the PRM-only rescue fraction from 13.65% to 5.88% on MMLU-Pro and from 14.00% to 6.80% on MATH500. To examine whether routing remains effective, we evaluated actual entropy-routing frontiers using the same aggregation rule on both internal and external branches.

BenchmarkAggregationExcess AUC [95% CI]Gain over FLOPs-matched random at τ=0.5\tau=0.5 (pp) [95% CI]
MMLU-ProBoN+0.0112 [0.0064, 0.0160]+0.829 [0.359, 1.297]
MMLU-ProWMV+0.0088 [0.0050, 0.0125]+0.505 [0.096, 0.973]
MATH500BoN+0.0070 [−0.0014, 0.0154]−0.227 [−1.195, 0.575]
MATH500WMV+0.0119 [0.0063, 0.0180]+0.552 [0.000, 1.400]

Excess AUC measures the full-frontier improvement over random routing matched by PRM call rate. The fixed-point comparison uses a no-choice-aware random baseline matched by expected PRM scoring FLOPs.

Under WMV, entropy routing retains positive excess AUC on both benchmarks. The paired changes in excess AUC from BoN to WMV are −0.0023 [−0.0074, 0.0027] on MMLU-Pro and +0.0049 [−0.0029, 0.0126] on MATH500, neither of which is statistically significant. Under the stricter FLOPs-matched comparison, the improvement is statistically supported on MMLU-Pro but not on MATH500.

We also evaluated Borda aggregation, obtaining smaller excess AUCs of 0.0026 and 0.0034, respectively. These results show that routing can remain beneficial beyond BoN, while its effectiveness depends on the aggregation rule and computational budget. We will extend the aggregation analysis to include these routing results.

Q1. Can scoring only a candidate subset improve the frontier?

Yes. We evaluated candidate pruning by selecting the top-kk candidates according to internal confidence and applying PRM scoring only to this subset (see Reviewer X749, W4-2). We also compared against randomly selected subsets and combined pruning with a free no-choice check that skips external scoring when all candidates yield the same answer. Computation is estimated using per-candidate token-weighted analytical FLOPs, including generation and PRM scoring.

BenchmarkNo-choice + Top-16: accuracy / total FLOPsFull PRM accuracyTop-16 − Full PRM (pp) [95% CI]Top-4 − Random-4 (pp) [95% CI]
MMLU-Pro60.74% / 68.99%60.79%−0.049 [−0.729, 0.583]+1.907 [0.665, 3.133]
MATH50064.20% / 74.03%64.40%−0.20 [−1.20, 0.80]+5.263 [2.988, 7.555]

Total FLOPs are normalized to full PRM scoring. No-choice + Top-16 achieves accuracy close to full PRM while reducing total computation by 31.01% on MMLU-Pro and 25.97% on MATH500. The accuracy differences from full PRM are not statistically significant, although this does not establish equivalence. At k=4k=4, confidence-based selection significantly outperforms random subset selection on both benchmarks, indicating that internal ranking is useful under smaller scoring budgets.

Compared with the original entropy gate, no-choice + Top-16 achieves higher observed accuracy with lower computation on both benchmarks. Candidate pruning therefore provides a stronger accuracy–compute operating point among the evaluated strategies. We will include these results as an additional baseline and clarify that the original binary routing policy studies question-level scoring-signal selection, whereas pruning also controls which candidates receive external scores. These comparisons are based on offline evaluations and analytical FLOPs rather than measured inference latency.

AuthorsOpenReview ↗

Gentle Follow-Up on Our Responses

Thank you again for your time and thoughtful comments on our manuscript. We have carefully addressed the concerns and questions raised in your review and provided detailed responses, including additional analyses and experimental results.

As the discussion period is approaching its end, we would like to kindly follow up in case you have not yet had a chance to review our responses. We appreciate your time and thoughtful feedback, and we hope our responses help clarify the concerns raised in your reviews.

Thank you again for your time and consideration.

Dear Reviewer DZv3,

As the discussion period nears its end, we would like to follow up on our rebuttal responses. We have provided detailed explanations and additional experimental results to address the concerns raised in your review, and we hope these have helped clarify the points you raised.

Thank you again for your time and thoughtful feedback.

Thank you for the response. My concerns are mostly addressed. I will maintain my positive rating.

Reviewer_YoE1OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies instance-level scoring signal selection for test-time scaling, where a selector determines whether to use an inexpensive internal confidence signal or invoke an external process reward model (PRM) to rank a fixed set of candidate solutions. The authors investigate the complementarity between internal and external scoring signals and evaluate several routing strategies, including training-free disagreement statistics and learned classifiers based on MLP and DeBERTa architectures. Experiments on MMLU-Pro and MATH500 show that answer entropy provides a useful heuristic for reducing PRM calls while approximately preserving external-scoring accuracy. However, the learned selectors do not consistently outperform this simple heuristic.

Critical strengths

The paper highlights the potential inefficiency of applying expensive external scoring uniformly across all reasoning instances. Selecting scoring signals after candidate generation is a reasonable direction for reducing unnecessary verification costs, and the analysis of complementary success cases provides useful motivation.

Critical weaknesses
  1. The proposed framework essentially applies a threshold-based routing rule to decide whether an external evaluator should be invoked. Using prediction disagreement or answer entropy as a proxy for uncertainty is well-established in adaptive inference and test-time scaling. Although the authors formulate the decision specifically as post-generation scoring signal selection, this distinction alone does not establish a substantial methodological advance. More importantly, the paper does not develop an effective new solution to the formulated problem. The strongest practical approach is a simple entropy threshold, while the proposed learned selectors generally fail to provide consistent improvements. Consequently, the main contribution is largely an empirical comparison of existing heuristics and standard classification architectures, rather than a novel routing methodology.

  2. The reported improvements are considerably less compelling than the motivating oracle analysis might suggest. In Table 1, the entropy heuristic achieves 60.50% accuracy on MMLU-Pro and 63.20% on MATH500, compared with 60.79% and 64.40% for always using the PRM. Thus, the method sacrifices accuracy while still invoking the external scorer on 71% and 73% of the instances, respectively. The reported total generation-plus-scoring FLOPs reductions are only 17.1% and 13.6%. While these savings are useful, they are relatively modest given that the proposed method is primarily motivated by computational efficiency.

Questions for authors

What is the fundamental methodological contribution beyond applying existing uncertainty or disagreement statistics to decide whether an external scorer should be invoked? How does the proposed entropy-based selector differ substantively from established uncertainty-guided adaptive inference strategies, apart from the specific placement of the routing decision?

5 comments
AuthorsOpenReview ↗

Response to Reviewer YoE1 (1/2)

W1-1 / Q1. What is the empirical contribution beyond uncertainty-triggered evaluation?

We appreciate the reviewer's question about the substantive contribution of our work. We agree that uncertainty-triggered evaluation is not new, and we do not claim the entropy threshold itself as an algorithmic contribution. Our study instead examines a specific decision: given a fixed set of generated reasoning trajectories, when does replacing the model's internal confidence scores with an external PRM improve answer selection? Unlike approaches that obtain additional reasoning or revise an existing solution, external scoring in our setting can only change which existing candidate is selected. This allows us to examine the benefits and limitations of changing scoring signals independently of further generation.

Our experiments provide three main findings.

First, scorer complementarity does not directly translate into predictable improvement. Self-Certainty and PRM produce weakly correlated candidate rankings (Kendall's τ = 0.177 on MMLU-Pro and 0.202 on MATH500), and the PRM uniquely selects the correct answer on 13.65% and 14.00% of questions under BoN. However, these opportunities are not simply concentrated on the most uncertain questions. On MATH500, the PRM-only success rate decreases from 19.1% at medium answer entropy to 12.6% at high entropy, where both scorers increasingly fail. Thus, uncertainty about the generated answers is not equivalent to the expected benefit of switching scorers.

Second, information about the current answer-selection decision helps predict this benefit. Our additional experiments show that whether the Self-Certainty BoN winner agrees with the majority of sampled answers provides useful information beyond answer entropy. Incorporating decision-level features into a logistic-regression selector improves excess AUC over the entropy heuristic by +0.0139 [0.0096, 0.0180] on MMLU-Pro and +0.0229 [0.0126, 0.0331] on MATH500. This also explains an important limitation of our original learned baselines: their representations did not explicitly capture the internal decision that external scoring would potentially change.

Third, scorer complementarity, predictability, and realizable computational benefit must be distinguished. Using free WMV instead of internal BoN reduces the PRM-only success opportunities, but does not eliminate the observed routing structure. After controlling for free no-choice skipping and matching expected PRM scoring FLOPs, the entropy gate retains a statistically supported advantage over random allocation on MMLU-Pro, but not on MATH500. Additional candidate-pruning experiments also identify more favorable observed accuracy–compute tradeoffs than the original binary gate. These controls show why the presence of complementary scoring signals alone is insufficient to establish the practical value of a particular routing policy.

Our new OlympiadBench evaluation further supports the relevance of this problem on more challenging reasoning tasks: PRM scoring improves accuracy under both BoN (+5.79 percentage points) and WMV (+2.23 points). The new decision-feature analyses are exploratory on previously examined test sets, and part of their BoN-specific improvement depends on the free aggregation baseline.

We will clarify the paper's contribution as a controlled empirical study of scoring-signal selection: identifying when external scoring offers additional value, what information helps anticipate that value, and which opportunities remain useful under stronger aggregation and compute-matched alternatives. Our contribution is this empirical characterization rather than a new or universally optimal uncertainty-based routing algorithm.

AuthorsOpenReview ↗

Response to Reviewer YoE1 (2/2)

W1-2. Interpretation of the original learned-selector results

The original learned-selector results require a more precise interpretation. Our MLP and DeBERTa baselines used question representations and aggregate response statistics but did not explicitly represent the answer selected by internal BoN or its relationship to competing answers. Their underperformance relative to entropy therefore does not establish that learned scoring-signal selection is ineffective.

To examine this, we evaluated additional decision-conditioned features. For internal BoN → PRM BoN routing, adding answer-distribution and internal-winner features to entropy improves logistic-regression excess AUC from 0.0112 to 0.0250 on MMLU-Pro and from 0.0070 to 0.0299 on MATH500. The paired improvements are +0.0139 [0.0096, 0.0180] and +0.0229 [0.0126, 0.0331], respectively (95% bootstrap CIs). Feature ablations show that internal-winner features account for most of the improvement, while answer-distribution statistics alone provide little gain.

We also examined training-data sensitivity. Using 25% of the training questions yields excess AUCs of 0.0244 and 0.0305, compared with 0.0250 and 0.0299 using the full training sets. This suggests that training-set size alone does not explain the original results.

We further evaluated whether the improvement persists with a stronger internal aggregation rule. A logistic selector retrained for internal WMV → PRM BoN achieves positive excess AUCs of 0.0118 and 0.0146, although the gains are smaller than under internal BoN. With WMV on both branches, a separate training-free winner–majority conflict gate does not significantly outperform entropy. This comparison does not evaluate the learned logistic selector under WMV-to-WMV routing.

We will revise the manuscript to clarify that the original results reflect the limitations of the evaluated feature representations rather than a general failure of learned routing. The additional experiments indicate that representing the internal selection decision can improve routing, while the remaining benefit depends on the aggregation baseline. We will present these analyses as exploratory results on previously examined evaluation sets.

W2. Practical significance of compute savings

We agree that the practical value of routing should be evaluated using total computation and its associated accuracy loss. We therefore recomputed the original entropy gate's cost using per-candidate token-weighted analytical FLOPs, including both generation and PRM scoring. At τ=0.5\tau=0.5, total FLOPs savings are 12.12% on MMLU-Pro and 6.56% on MATH500, compared with the original average-cost estimates of approximately 17.1% and 13.6%. Relative to always using the full PRM, the accuracy differences are −0.292 percentage points [−0.777, 0.146] and −1.20 percentage points [−2.20, −0.40], respectively (95% paired bootstrap CIs). The accuracy loss is statistically detectable on MATH500 but not on MMLU-Pro. These FLOPs estimates do not represent measured inference latency.

We also evaluated candidate pruning as an alternative computation allocation strategy (see Reviewer X749, W4-2). In particular, we applied a free no-choice check followed by PRM scoring of the 16 candidates with the highest internal confidence.

BenchmarkEntropy gateNo-choice + Top-16Full PRM
MMLU-Pro60.50% / 87.88%60.74% / 68.99%60.79% / 100%
MATH50063.20% / 93.44%64.20% / 74.03%64.40% / 100%

Each entry reports accuracy and total FLOPs normalized to always using the full PRM. No-choice + Top-16 achieves higher observed accuracy with lower computation than the original entropy gate on both benchmarks, although these comparisons do not establish statistically significant dominance.

These results clarify the practical scope of our approach. The original binary routing policy evaluates whether external scoring is needed for a given question, whereas candidate pruning additionally determines which candidates should receive external scores. Our experiments show that question-level routing can reduce scoring computation, but its efficiency gains depend on the benchmark and are not necessarily optimal among available allocation strategies. We will update the cost analysis and clarify this distinction in the revised manuscript.

AuthorsOpenReview ↗

Gentle Follow-Up on Our Responses

Thank you again for your time and thoughtful comments on our manuscript. We have carefully addressed the concerns and questions raised in your review and provided detailed responses, including additional analyses and experimental results.

As the discussion period is approaching its end, we would like to kindly follow up in case you have not yet had a chance to review our responses. We appreciate your time and thoughtful feedback, and we hope our responses help clarify the concerns raised in your reviews.

Thank you again for your time and consideration.

Reviewer_X749OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper investigates the problem of post-generation scoring signal selection in Best-of- (BoN) sampling for Test-Time Scaling (TTS). Given a question and generated candidate solutions , the core objective is to dynamically decide per instance whether to rank candidates using a zero-additional-cost internal confidence signal (Self-Certainty, ) or to invoke a computationally expensive external Process Reward Model (PRM, ) to rescore all candidates. Through empirical evaluations on MMLU-Pro and MATH500, the authors demonstrate that internal and external scoring signals exhibit instance-level complementarity. Furthermore, they find that simple training-free candidate disagreement statistics (such as normalized answer entropy ) capture coarse-grained routing utility, whereas learned selectors—including MLPs over hidden-state representations and fine-tuned DeBERTa models—fail to consistently outperform simple scalar answer entropy across the accuracy–call-rate Pareto frontier.

Critical strengths
  1. Clean Problem Formulation and Practical Motivation: Unlike pre-generation model routing or adaptive early-stopping sampling, this work focuses on the decision stage after candidate generation but before invoking an expensive external verifier. Since all candidate trajectories have already been generated at this point, leveraging zero-cost internal candidate-set statistics to avoid unnecessary 7B/72B PRM forward passes is logically natural, highly modular, and easy to integrate into existing TTS inference pipelines.
  2. Well-Crafted Figures and Intuitive Visual Exposition: The visual presentation of the paper is of high quality, with well-designed color schemes, layouts, and information density that substantially enhance readability. Specifically, Figure 1 and Figure 2 clearly illustrate the motivation, workflow, and ranking discrepancies between internal and external signals; meanwhile, the Pareto frontier comparisons (Figure 3), outcome trends across entropy deciles (Figure 4) in the main text, and the multi-faceted visualizations in the appendix (such as the aggregation rule bar charts in Figure 6, hexbin density plots in Figure 7, and cross-dataset paired ablation plots in Figures 8/10) effectively convey complex multi-dimensional empirical patterns at a glance.
  3. Clear Analysis of the Underlying Mechanism and Failure Modes of Entropy Routing: In Section 4.3 (Figure 4 and Table 3), the authors clearly reveal the underlying mechanism of answer entropy as a coarse-grained routing signal: in the low-entropy regime, the internal signal is already highly reliable ("Both correct" dominates); in the medium-entropy regime, the PRM-only rescue rate rises; and in the highest-entropy regime, because "Neither correct" dominates, the rescue benefit of the PRM declines again. This non-monotonicity intuitively explains why the entropy heuristic falls below random routing at very low call rates (high thresholds) on MATH500.
Critical weaknesses
  1. Insufficient Evaluation Set: Evaluations are conducted only w.r.t. MMLU-Pro and MATH 500, which are not challenging and representative at this era of LLM development.
  2. Fragile Baseline Setup and Limited Practical Routing Gains: The main text restricts the aggregation rule to unaggregated trajectory-level BoN, which artificially amplifies the vulnerability of the internal signal; however, Appendix A.2 (Figure 6) shows that simply applying Weighted Majority Voting (WMV, at zero external cost) to the free internal confidence signal boosts internal-only accuracy by on MMLU-Pro and on MATH500, while cutting the unique PRM-only rescue margin roughly in half (e.g., from down to on MATH500). Moreover, the saved PRM calls include a substantial fraction of trivial instances where all 32 candidates yield the exact same answer (, requiring no scorer at all; Footnote 1, Figure 7). Even with these trivial savings included, the Excess AUC of the entropy heuristic over random routing in the primary MATH500 (7B PRM) setting remains statistically insignificant (95% CI overlaps zero, Section 4.4), and total generation-plus-scoring FLOPs are reduced by only 13.6%–17.1% at the expense of accuracy drops, yielding very modest practical efficiency gains.
  3. Crude Feature Design for Learned Selectors Does Not Support the Claim that Fine-Grained Routing is Infeasible: The paper claims that learned selectors cannot capture fine-grained routing signals beyond scalar answer entropy, but this directly stems from feature engineering that discards the most critical decision-level information. DeBERTa is completely blind to the candidate set, while the MLP simply applies unweighted mean-pooling (R-Rep) over the hidden states of all 32 candidates, completely washing out the specific representation of the top-1 candidate selected by the internal signal. Furthermore, the models are neither fed the confidence distribution across candidates and the margin to the runner-up answer cluster, nor given the crucial conflict feature indicating whether the internal BoN winning answer disagrees with the majority-vote answer. Given that the models have no information about which trajectory the internal scorer picked or how robust that pick is, the degradation of binary classification ROC-AUC to near-random guessing at (Table 6; for MLP and for DeBERTa on MATH500) is an artifact of feature design and cannot be used to conclude that fine-grained routing is inherently unlearnable.
  4. Test-Set Selection Leakage in Evaluation and Overly Rigid "All-or-Nothing" Action Space: The experimental pipeline has methodological flaws in evaluation rigor: Appendix A.8 admits that the MLP (best) and DeBERTa (best) entries reported in Table 1 are selected post-hoc from 11 configurations based directly on the highest test-set accuracy (and the chosen best MLP on MATH500 is actually the degenerate univariate Entropy-only variant, whereas adding hidden-state features causes accuracy to drop from to ), while the multi-seed stability script (Table 7) uses a different early-stopping criterion from the main Table 1 script, preventing exact numeric reproduction. In addition, the paper rigidly restricts the routing action space to an instance-level "all-or-nothing" choice (invoking the PRM on all 32 candidates or on 0), overlooking natural candidate-level pruning strategies that use internal confidence to filter out low-scoring trajectories or deduplicate by answer cluster to invoke the PRM only on the top- representative candidates.
Questions for authors

Why is unaggregated raw BoN used as the primary internal baseline when free Weighted Majority Voting (WMV, shown in Appendix A.2) significantly boosts internal accuracy and reduces the PRM-only rescue margin? How would the Pareto frontiers change if routing were performed between Internal WMV and External PRM?

9 comments
AuthorsOpenReview ↗

Response to Reviewer X749 (1/6)

W1. Benchmark difficulty and representativeness

We appreciate the reviewer's concern about benchmark difficulty and representativeness. Our study focuses on choosing between scoring signals over a fixed set of generated reasoning trajectories. For this purpose, an informative benchmark should contain correct candidates that a scorer could select, leave room for improving answer selection, and exhibit differences between the scoring signals. The original MMLU-Pro and MATH500 settings satisfy these conditions, with pass@32 of 89.26% and 77.40%, while Self-Certainty BoN achieves 53.21% and 53.20%, respectively.

To examine whether our findings extend to more challenging reasoning tasks, we conducted additional experiments on OlympiadBench and OlymMATH, using MATH500 as a reference under the same model configuration. All four evaluations use Qwen2.5-Math-7B-Instruct as the generator, Qwen2.5-Math-PRM-7B as the external scorer, N=32 sampled candidates, and minimum step-score aggregation for the PRM. We report both Best-of-N (BoN) and weighted majority voting (WMV), the latter providing a stronger aggregation baseline.

BenchmarkQuestionsPass@32 (%)Self-Certainty BoN (%)PRM BoN (%)Self-Certainty WMV (%)PRM WMV (%)
MATH50050093.8085.6086.8086.4087.00
OlympiadBench67471.0745.4051.1950.0052.23
OlymMATH-EASY10035.0012.0013.0011.0012.00
OlymMATH-HARD10018.002.004.004.004.00

On OlympiadBench, we observe the main scorer-complementarity phenomenon found in our original experiments. The two scorers rank candidates differently (Kendall's τ = 0.155), and PRM BoN selects a correct answer on 66 questions where Self-Certainty BoN fails, compared with 27 cases in the opposite direction. This yields a 5.79 percentage-point improvement [2.97, 8.46]. Importantly, the PRM advantage persists under WMV, with an improvement of 2.23 points [0.74, 3.86]. Both intervals are 95% paired question-bootstrap confidence intervals.

We also observe some predictability of this benefit. At the original entropy threshold of 0.5, routing outperforms a no-choice-aware, matched-call-rate random baseline by 0.98 points [0.18, 1.79] on OlympiadBench. However, the advantage is not statistically established when matching expected PRM scoring FLOPs. Thus, the new results support the presence of scorer-specific selection opportunities, while distinguishing them from the performance of a particular routing policy.

OlymMATH provides a complementary observation at higher difficulty. With pass@32 of only 35% on EASY and 18% on HARD, most questions lack a correct candidate for either scorer to select. The observed scorer differences are correspondingly small and statistically inconclusive. This highlights candidate availability as an important constraint on scoring-signal selection.

We will incorporate these additional evaluations and clarify the rationale for our benchmark choices in the revised paper. The new results extend our empirical observations to competition-level mathematics and identify a limitation on harder problems, without implying generalization to all reasoning domains or larger models.

AuthorsOpenReview ↗

Response to Reviewer X749 (2/6)

W2-1 / Q1. Why BoN rather than weighted majority voting?

We appreciate the reviewer's suggestion to consider weighted majority voting (WMV) as a stronger internal baseline. We originally used Best-of-N (BoN) to isolate the effect of changing scoring signals while keeping the aggregation rule fixed. However, WMV improves answer selection using the same internal scores without additional PRM computation. On MMLU-Pro and MATH500, replacing Self-Certainty BoN with Self-Certainty WMV increases accuracy from 53.21% to 57.48% and from 53.20% to 58.40%, respectively. Correspondingly, the fraction of questions where only the PRM selects the correct answer decreases from 13.65% to 5.88% on MMLU-Pro and from 14.00% to 6.80% on MATH500 when both scorers use WMV rather than BoN. We therefore agree that the original BoN-based rescue opportunity depends on the aggregation rule and should also be evaluated against the stronger free WMV baseline.

We conducted additional experiments with two comparisons: Self-Certainty WMV versus PRM BoN, which directly evaluates the original external-scoring action against a stronger internal baseline, and Self-Certainty WMV versus PRM WMV, which isolates the effect of changing scoring signals while keeping aggregation fixed. We also include the original BoN comparison for reference. All experiments use the same N=32 candidate pools and the original entropy gate at τ=0.5.

BenchmarkInternal → PRM aggregationRouting accuracy at τ=0.5 (%)Pareto-AUCExcess AUC (95% CI)Gain over cost-matched random, pp (95% CI)
MMLU-ProBoN → BoN60.500.5810+0.0112 [0.0064, 0.0160]+0.829 [0.359, 1.297]
MMLU-ProWMV → BoN60.980.6013+0.0100 [0.0053, 0.0150]+0.676 [−0.049, 1.409]
MMLU-ProWMV → WMV60.930.6009+0.0088 [0.0050, 0.0125]+0.505 [0.096, 0.973]
MATH500BoN → BoN63.200.5950+0.0070 [−0.0014, 0.0154]−0.227 [−1.195, 0.575]
MATH500WMV → BoN63.800.6232+0.0092 [0.0028, 0.0161]−0.095 [−1.000, 0.800]
MATH500WMV → WMV64.800.6279+0.0119 [0.0063, 0.0180]+0.552 [0.000, 1.400]

Pareto-AUC summarizes the accuracy–PRM-call-rate frontier over the entropy threshold sweep. Excess AUC measures the area above matched-call-rate random allocation. The final column reports the accuracy improvement at τ=0.5 over a no-choice-aware random baseline matched in expected token-weighted PRM scoring FLOPs. All intervals are 95% paired question-bootstrap confidence intervals.

Using WMV on both branches raises Pareto-AUC from 0.5810 to 0.6009 on MMLU-Pro and from 0.5950 to 0.6279 on MATH500, partly reflecting the stronger internal aggregation baseline. Importantly, the routing advantage over matched-call-rate random allocation remains positive under WMV. The paired changes in excess AUC relative to the original BoN-to-BoN setting are −0.0023 [−0.0074, 0.0027] and +0.0049 [−0.0029, 0.0126], respectively. Neither change is statistically established, so the results do not indicate a detectable loss of routing advantage when switching to WMV, although they do not establish equivalence.

Under the more stringent FLOPs-matched random comparison, the advantage is statistically supported for WMV-to-WMV routing on MMLU-Pro, but not on MATH500 or for the mixed WMV-to-BoN setting. The new OlympiadBench evaluation provides further evidence that external scoring retains value under stronger aggregation: PRM WMV achieves 52.23% accuracy versus 50.00% for Self-Certainty WMV, a gain of +2.23 percentage points [0.74, 3.86]. This establishes an average PRM selection advantage, although it does not by itself demonstrate a corresponding routing advantage.

These results show that stronger free aggregation substantially reduces the original PRM-only rescue opportunities, but does not eliminate the observed scoring-signal differences or their predictability under entropy routing. We will include the WMV-based routing comparisons in the revised paper and distinguish the effect of changing scoring signals from that of changing aggregation rules.

AuthorsOpenReview ↗

Response to Reviewer X749 (3/6)

W2-2. Free no-choice cases and avoidable PRM calls

Some PRM evaluations can be skipped without adaptive routing. We define a no-choice case as a question with at least one valid candidate but only one distinct valid answer. Under our answer-based aggregation rules and invalid-answer masking, all scoring methods select the same answer in such cases, so running the PRM cannot change the final prediction.

We quantified how much of the original entropy gate's savings can be explained by this simple check. The following results use the original N=32 candidate pools.

BenchmarkNo-choice questionsShare of entropy-skipped callsPRM scoring FLOPs saved by no-choice alonePRM scoring FLOPs saved by entropy routing
MMLU-Pro232 / 2,05838.4%7.21%20.92%
MATH50061 / 50044.9%4.86%13.13%

A substantial fraction of the skipped calls can indeed be avoided without a learned or heuristic routing decision. However, these no-choice questions tend to have shorter PRM inputs, so their contribution to the saved token-weighted scoring FLOPs is smaller than their share of skipped calls.

To determine whether entropy routing provides additional value beyond this free optimization, we evaluated random allocation strategies that also skip no-choice questions. We compared both matched PRM call rates and matched expected token-weighted PRM scoring FLOPs.

Random baselineMMLU-Pro: entropy − random (pp)MATH500: entropy − random (pp)
No-choice-aware, matched call rate+1.253 [0.740, 1.765]+0.713 [−0.311, 1.623]
No-choice-aware, expected PRM-FLOPs-matched+0.829 [0.359, 1.297]−0.227 [−1.195, 0.575]

The values are accuracy differences at the original entropy threshold of 0.5, with 95% paired question-bootstrap confidence intervals. The FLOPs-matched comparison controls expected PRM scoring cost rather than the number of calls.

After accounting for free no-choice skipping, entropy routing retains a statistically supported instance-selection advantage on MMLU-Pro, including under the FLOPs-matched comparison. On MATH500, the additional advantage is not statistically established. We will distinguish savings from free no-choice checks and adaptive routing in the revised paper, and evaluate routing gains against baselines that receive the same free optimization.

W2-3. How much computation does routing actually save?

PRM call reduction does not directly translate into FLOPs reduction for two reasons. First, candidate lengths vary across questions, and the questions skipped by our entropy gate tend to require less PRM computation. Second, all 32 candidates must still be generated before the routing decision is made, so generation cost cannot be avoided.

Our original manuscript estimated total FLOPs savings using average generation and scoring costs. We have now recomputed the same entropy policy at the original threshold τ = 0.5, using per-candidate token-weighted analytical FLOPs. The updated calculation accounts for the actual lengths of candidates scored by the PRM and includes the generation cost of all 32 candidates. We normalize both PRM scoring FLOPs and total generation-plus-scoring FLOPs against always scoring all 32 candidates with the PRM.

BenchmarkPRM call rate (%)PRM scoring FLOPs (% of full)Total FLOPs (% of full)Total FLOPs saved (%)Entropy − full PRM accuracy (pp, 95% CI)
MMLU-Pro70.6579.0887.8812.12−0.292 [−0.777, 0.146]
MATH50072.8086.8793.446.56−1.20 [−2.20, −0.40]

The accuracy differences are evaluated at τ = 0.5 using 95% paired question-bootstrap confidence intervals. On MMLU-Pro, entropy routing achieves 60.50% accuracy compared with 60.79% for always-full PRM, saving 12.12% of total FLOPs without a statistically detectable accuracy loss. On MATH500, it achieves 63.20% versus 64.40%, saving 6.56% of total FLOPs with a statistically supported 1.20 percentage-point accuracy loss.

The updated total savings are lower than the original estimates of approximately 17.1% on MMLU-Pro and 13.6% on MATH500 because the new calculation accounts for individual candidate lengths rather than using average scoring costs. These are analytical FLOPs estimates, not measurements of GPU latency or serving throughput.

The results confirm that question-level routing can reduce external scoring computation, but its practical accuracy–compute benefit is benchmark-dependent. Our additional candidate-pruning experiments also identify more favorable observed accuracy–compute operating points than the original entropy gate. We will update the computational analysis and clarify that the binary routing policy serves as a controlled setting for studying scoring-signal selection, rather than an optimized deployment strategy.

Reviewer_A1XBOpenReview ↗
Rating
2
Confidence
3(+1)
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

The paper studies whether to use internal confidence or an external reward model to select answers from a fixed set of generated candidates. It evaluates disagreement-based heuristics and learned gates, finding that simple answer-entropy routing offers useful accuracy–cost tradeoffs, while learned selectors do not consistently improve upon it.

Critical strengths

The paper clearly presents a practical scoring-cost problem and provides extensive comparisons, ablations, and statistical analyses.

Critical weaknesses
  1. The novelty appears primarily to lie in a narrower experimental setting. ACR (§3.2) already uses candidate disagreement to trigger additional evaluation, while SeVRA (§6.3, Table 3) finds inexpensive feature gates competitive with larger learned gates. Fixed-candidate switching between internal confidence and an external PRM is distinct, but the paper should clarify what substantial new insight this specialization provides.

  2. The gate’s additional benefit over matched-budget random routing is modest. Using the paper’s formula Arandom(c)=(1−c)AI+cAEA_{\mathrm{random}}(c)=(1-c)A_I+cA_E, the following calculations separate the benefit of invoking the stronger PRM from the benefit of selecting which questions receive it. Random accuracies are approximate expectations based on the rounded values in Table 1; gains are in percentage points.

    For MMLU-Pro:

    MethodPRM call rateAccuracyMatched-rate randomGain over random
    Internal-only0%53.21%53.21%0.00
    PRM-only100%60.79%60.79%0.00
    Confidence heuristic97%60.88%60.56%+0.32
    Entropy heuristic71%60.50%58.59%+1.91
    MLP (best)71%59.52%58.59%+0.93
    DeBERTa (best)95%60.16%60.41%−0.25
    Oracle*14%66.86%54.27%+12.59

    For MATH500:

    MethodPRM call rateAccuracyMatched-rate randomGain over random
    Internal-only0%53.20%53.20%0.00
    PRM-only100%64.40%64.40%0.00
    Confidence heuristic94%64.60%63.73%+0.87
    Entropy heuristic73%63.20%61.38%+1.82
    MLP (best)65%62.40%60.48%+1.92
    DeBERTa (best)39%57.60%57.57%≈0
    Oracle*14%67.20%54.77%+12.43

    The oracle requires correctness information and is not deployable.

    Most improvement over internal-only scoring is therefore already obtainable through random allocation. These individual operating-point differences have no reported pointwise significance tests. Table 11 establishes significant full-frontier gains on MMLU-Pro and three alternative MATH500 scorers, but not the primary MATH500 configuration.

  3. The extensive analysis does not yet adequately explain why learned selection adds so little value. Applying the same calculation to Table 2 shows that richer representations frequently provide smaller gains over random routing. “Accuracy / calls” below reports accuracy and PRM call rate, both as percentages.

    MLP inputMMLU accuracy / callsRandomGainMATH accuracy / callsRandomGain
    Entropy-only59.23 / 5957.68%+1.5562.40 / 6560.48%+1.92
    Response representation53.40 / 653.66%−0.2661.20 / 8062.16%−0.96
    Response representation + entropy + std.56.41 / 3455.79%+0.6256.00 / 2355.78%+0.22
    Question representation56.17 / 4256.39%−0.2257.00 / 2956.45%+0.55
    Question representation + entropy59.52 / 7158.59%+0.9357.00 / 2856.34%+0.66

    Here, “Entropy-only” is a trained MLP, rather than the threshold heuristic. These point estimates do not establish significant differences. Appendix A.11 measures candidate-subset sensitivity but explicitly does not connect it to learned-selector performance. A targeted diagnosis separating representation and training-data limitations from candidate-dependent outcome variability would make the negative finding more informative beyond the evaluated configurations.

Questions for authors
  1. What substantial new insight does fixed-candidate internal-score/PRM switching provide beyond ACR’s disagreement-triggered evaluation and SeVRA’s selective verification?

  2. Using unrounded call rates, what are the accuracy gains and paired confidence intervals over matched-budget random routing? Are the approximately 1.91/1.82-point gains significant at the reported operating points, separately from the full-frontier tests?

  3. Why do learned gates fail to consistently outperform simple entropy routing? What evidence distinguishes limitations of training data or representations from candidate-dependent outcome variability?

9 comments
AuthorsOpenReview ↗

Response to Reviewer A1XB (1/2)

W1 / Q1. Contribution relative to ACR and SeVRA

Thank you for raising this comparison. We agree that selectively invoking additional evaluation is not new. While ACR obtains additional option-selection feedback and SeVRA selectively verifies or revises an existing attempt, our study focuses on a different decision: given a fixed set of generated reasoning trajectories, when is it beneficial to replace the model's own confidence scores with an external PRM for answer selection? By keeping the candidates unchanged, we can examine the value of switching scoring signals independently of further generation or revision.

This controlled setting leads to three empirical findings.

First, the two scoring signals provide complementary information. Their rankings of the same candidates have low Kendall correlations (0.177 on MMLU-Pro and 0.202 on MATH500). More importantly, the PRM selects a correct answer when Self-Certainty BoN selects an incorrect one on 13.65% and 14.00% of questions, respectively. These cases establish opportunities for improving answer selection by changing the scorer.

Second, the benefit of external scoring is not simply determined by how uncertain the model is. On MATH500, the fraction of questions where only the PRM selects the correct answer decreases from 19.1% at medium answer entropy to 12.6% at high entropy, where both scorers fail more frequently. Thus, high uncertainty does not necessarily imply that an additional scorer will improve the answer.

Third, we further investigated what information could help predict when switching to the PRM would improve answer selection. Our new feature analysis shows that the answer selected by Self-Certainty BoN, particularly whether it agrees with the majority of sampled answers, provides useful information beyond answer entropy. Adding these features to a logistic-regression selector improves excess AUC from 0.0112 to 0.0250 on MMLU-Pro and from 0.0070 to 0.0299 on MATH500. These results are exploratory and specific to the evaluated BoN setting, where part of the improvement can also be attributed to the stronger free aggregation alternative.

The central question is therefore not simply how uncertain the model is, but whether an external PRM is likely to improve the answer selected by Self-Certainty. Our experiments characterize when such improvements occur and what information is useful for anticipating them. We will make this distinction clearer in the revised paper, emphasizing the empirical findings rather than presenting the entropy threshold as a novel selective-evaluation algorithm.

W2 / Q2. Improvement over matched-budget random allocation

We thank the reviewer for suggesting a matched-budget random baseline. The key question is whether the entropy gate identifies questions that benefit more from external PRM scoring, rather than simply gaining accuracy by invoking the stronger scorer more frequently.

At the original threshold of τ = 0.5, the PRM call rates are 70.65% on MMLU-Pro and 72.80% on MATH500. Compared with randomly selecting the same proportion of questions for PRM scoring, our entropy gate achieves accuracy improvements of +1.933 and +1.846 percentage points (pp), respectively. Both improvements are statistically significant under paired question-level bootstrap tests.

We further evaluated two stronger random baselines. The first allows random routing to skip questions with only one distinct valid answer, where external scoring cannot change the final answer. The second additionally matches the expected token-weighted PRM scoring FLOPs, accounting for differences in candidate lengths. The results at τ = 0.5 are summarized below.

Random allocation baselineMMLU-Pro ΔAcc (pp)MATH500 ΔAcc (pp)
Matched PRM call rate+1.933 [1.320, 2.567]+1.846 [0.646, 2.976]
No-choice-aware, matched call rate+1.253 [0.740, 1.765]+0.713 [−0.311, 1.623]
No-choice-aware, expected PRM-FLOPs-matched+0.829 [0.359, 1.297]−0.227 [−1.195, 0.575]

Here, ΔAcc denotes the accuracy of entropy routing minus that of random routing, with 95% confidence intervals from 2,000 paired bootstrap resamples over questions. The FLOPs-matched baseline controls expected PRM scoring cost rather than the number of PRM calls.

The entropy gate therefore provides a measurable instance-selection advantage over the requested matched-call-rate random baseline on both datasets. After accounting for free no-choice skipping and PRM scoring cost, this advantage remains statistically supported on MMLU-Pro, but is not established on MATH500. We will include these fixed-threshold comparisons and their confidence intervals in the revised paper.

AuthorsOpenReview ↗

Response to Reviewer A1XB (2/2)

W3 / Q3. Why do learned selectors not consistently outperform entropy routing?

We conducted additional experiments to examine the roles of input features, training data, and candidate-set variability.

The question-only DeBERTa and pooled-response MLP inputs do not explicitly identify the answer selected by Self-Certainty BoN from the 32 candidates. We therefore tested two groups of features available before PRM scoring. Answer-consensus statistics summarize the fraction of candidates supporting the most frequent answer, the support margin between the two leading answers, and the number of distinct answers. Self-Certainty winner features record the highest-scoring candidate's confidence, its margin over the second-ranked candidate, and whether its answer differs from unweighted majority voting. This conflict directly describes the current selection decision, rather than only overall uncertainty.

We trained logistic-regression (LogReg) and gradient-boosted-tree (GBDT) routers using the original question-level splits and evaluated actual routing frontiers between Self-Certainty BoN and PRM BoN.

BenchmarkRouter and featuresValidation ROC-AUCTest Pareto-AUCTest excess AUC
MMLU-ProEntropy heuristic (no training)—0.5810+0.0112
MMLU-ProLogReg: entropy only0.68950.5810+0.0112
MMLU-ProLogReg: entropy + answer consensus0.68210.5805+0.0107
MMLU-ProLogReg: entropy + answer consensus + winner features0.72470.5948+0.0250
MMLU-ProGBDT: same full feature set0.72650.5887+0.0189
MATH500Entropy heuristic (no training)—0.5950+0.0070
MATH500LogReg: entropy only0.63040.5950+0.0070
MATH500LogReg: entropy + answer consensus0.65020.5979+0.0099
MATH500LogReg: entropy + answer consensus + winner features0.81760.6179+0.0299
MATH500GBDT: same full feature set0.85970.6231+0.0351

Pareto-AUC summarizes the accuracy–PRM-call-rate frontier, and excess AUC measures its area above matched-call-rate random allocation. Thus, the table reports the trained routers' routing performance, not just their classification scores. Adding winner features improves LogReg excess AUC over the entropy heuristic by +0.0139 [0.0096, 0.0180] on MMLU-Pro and +0.0229 [0.0126, 0.0331] on MATH500 (95% paired question-bootstrap CIs). Answer-consensus features alone add little. A finer ablation finds that the winner–majority disagreement indicator alone achieves +0.0251 and +0.0252 excess AUC, explaining much of the improvement.

Training-size and candidate-set analyses help interpret the result. With 25% versus 100% of training questions, the full LogReg reaches +0.0244 versus +0.0250 excess AUC on MMLU-Pro and +0.0305 versus +0.0299 on MATH500 (25% averaged over three subsamples and one run at 100%). Training size alone therefore does not explain the original shortfall in these settings. Across 20 sampled subsets per question from cached 64-candidate pools, the preferred scorer changes on 12.10% and 5.20% of questions at N=32. Nevertheless, the within-question correlation between winner–majority disagreement and PRM-minus-Self-Certainty correctness is positive: +0.312 [0.270, 0.352] and +0.560 [0.481, 0.642], respectively, among questions where both vary. Candidate-set variability is therefore not entirely uninformative.

We also checked whether the improvement merely identifies cases where free weighted majority voting (WMV) would outperform internal BoN. Using internal WMV as the baseline gives the following routing results:

Scoring comparisonRouterMMLU-Pro excess AUCMATH500 excess AUC
Internal WMV → PRM BoNLogReg retrained for this comparison+0.0118+0.0146
Internal WMV → PRM BoNFixed winner–majority conflict gate+0.0102 [0.0053, 0.0151]+0.0088 [0.0015, 0.0158]
Internal WMV → PRM WMVFixed winner–majority conflict gate+0.0080 [0.0044, 0.0115]+0.0111 [0.0046, 0.0175]

The fixed conflict gate compares the Self-Certainty BoN answer with unweighted majority voting. It differs from the trained LogReg. The reduced gains show that part of the original BoN result relates to free aggregation, although the conflict signal retains some scorer-relative information after WMV. Under same-rule WMV, this fixed gate is not significantly better than entropy, and its advantage over no-choice-aware PRM-FLOPs-matched random routing is not established on MATH500.

We will revise the paper's interpretation of the original negative result. The initial learned baselines omitted information about the answer decision that PRM scoring might change. Their performance does not establish that learned scorer selection is ineffective in general. The new results are exploratory on previously examined test sets, and their implications depend on the free aggregation baseline.

AuthorsOpenReview ↗

Gentle Follow-Up on Our Responses

Thank you again for your time and thoughtful comments on our manuscript. We have carefully addressed the concerns and questions raised in your review and provided detailed responses, including additional analyses and experimental results.

As the discussion period is approaching its end, we would like to kindly follow up in case you have not yet had a chance to review our responses. We appreciate your time and thoughtful feedback, and we hope our responses help clarify the concerns raised in your reviews.

Thank you again for your time and consideration.

Comments on the paper

1 comment

We evaluated questions where Self-Certainty BoN and unweighted majority voting (MV) select different answers, using the same original N=32 candidate pools. The table reports all three methods' accuracies on this identical subset, along with the numbers of majority-vote errors corrected and correct majority-vote predictions made incorrect by PRM BoN.

BenchmarkDisagreement questionsSC-BoN acc.MV acc.PRM BoN acc.MV errors fixed by PRMCorrect MV answers flipped by PRM
MMLU-Pro642 / 2,058 (31.20%)19.78%34.11%43.46%13070
MATH500186 / 500 (37.20%)12.90%25.81%37.63%297

On these disagreement cases, unweighted MV improves upon Self-Certainty BoN by +14.33 pp [8.94, 19.66] on MMLU-Pro and +12.90 pp [4.42, 21.21] on MATH500. Thus, free majority voting corrects part of the weakness of internal BoN in these cases.

PRM BoN improves further over unweighted MV by +9.35 pp [5.09, 13.55] on MMLU-Pro and +11.83 pp [6.04, 18.03] on MATH500. It corrects 130 majority-vote errors while making 70 previously correct majority-vote predictions incorrect on MMLU-Pro. The corresponding counts on MATH500 are 29 and 7. Both PRM-over-MV gains are statistically supported by 2,000 paired question-bootstrap resamples, with the disagreement condition recomputed within each resample.

On the disagreement subset, free majority voting improves accuracy over Self-Certainty BoN, while PRM scoring provides a further, statistically supported improvement beyond majority voting. We will clarify this distinction in the revised discussion of decision-conditioned features.

Thank you again for your constructive feedback. We hope the additional experiments have addressed your remaining concerns, and we would appreciate it if you could reconsider your assessment based on these new results.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.