acceptodds
Under review as a conference paper at ICLR 2027

A Score Is Not a Selector: From Prediction to Branch Selection at Test Time

16 views

Abstract

Process reward models (PRMs) are benchmarked on whether they can tell which reasoning steps are correct, yet test-time search uses them to choose which branch to continue. Using the official step labels of PRM800K and ProcessBench as a checker that knows exactly which steps are valid, we find that even this checker loses to four to eight sampled Qwen3-4B-Instruct continuations per branch graded against the reference answer, by 1.71 and 2.65 points of final correctness, mostly because labels cannot rank branches that share a label. We therefore let sampled continuations rank the branches and consult a PRM only on the remaining ties. For reference-graded continuations, we prove that one more continuation per branch is worth exactly its expected gain from breaking the current top ties, so a PRM on these ties can be priced in continuations. On ProcessBench, Qwen-PRM's tie-breaking is worth about one continuation per branch when continuations are few. By eight continuations its gain falls to about zero on both datasets, and the gain from official labels shrinks eightfold. A synthetic example shows that even a perfect checker can hurt on ties. A process score should therefore be judged by what it adds beyond sampled continuations. In a reference-free search that ranks branches by self-consistency, PRM tie-breaking adds 0.16 points per selection point.

Primary area:
foundation or frontier models, including LLMs
Keywords:
process reward models, language model reasoning, test-time scaling, reasoning evaluation

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_GdcyOpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

This paper studies chain-of-thought branch selection, where process reward models (PRMs) are commonly used to favor valid reasoning steps. The authors argue that selection should also consider how promising each branch is for reaching a correct final answer. They therefore consider bounded recoverability, the probability that a given model completes a branch correctly within a specified token budget.

The authors compare selection using reference-graded continuations, which are checked against known answers, with reference-free selection based on agreement among sampled answers. Their experiments show that step validity alone is insufficient for identifying promising branches. They further use PRMs to break ties between branches with equal continuation-based scores. The results suggest that PRMs can provide useful additional guidance in this role, particularly when only a few continuations are available, while their contribution generally decreases as sampling increases.

Critical strengths
  • The paper highlights recoverability as an important consideration in branch selection, showing that a model can still reach a correct final answer even when an intermediate reasoning step is invalid.
Critical weaknesses
  • The practical applicability of the main MC-based selection analysis is limited by its reliance on reference answers. Although acknowledged in Section 8, this is a substantial limitation because correct answers are generally unavailable when solving new problems at test time. Gold-answer verification supplies information directly aligned with the final-correctness metric, so the resulting advantage over validity-based selection does not by itself establish a practical advantage over PRMs. The authors also evaluate a reference-free consensus variant, but agreement among sampled answers can favor a common wrong answer. For example, on OpenR1, the consensus answer is frequently incorrect, and the resulting accuracy improvement over uniform selection remains uncertain (Table 18). The paper should more clearly distinguish the conclusions supported by reference-graded experiments from those established without reference answers, and provide stronger evidence for the reliability of the reference-free approach.
  • I found parts of the results section difficult to follow. Section 4.1 moves quickly between results on different datasets and tasks, with little explanation of what they establish together. Section 6 similarly moves from tie-breaking experiments to comparisons with extra rollouts and then to sequential search. The paper provides explanations and takeaways, but the many numerical comparisons make the main argument harder to follow. I would like the authors to explain more clearly why each analysis follows from the previous one and discuss what the results collectively tell us about when PRMs are useful.
  • The study of combining PRM scores with rollout evidence focuses on tie-breaking. This isolates the PRM's contribution, but does not establish whether restricting it to exact ties is the most effective use of its scores. With only a few rollouts, small differences between sampled scores may be unreliable, and PRM scores could still provide useful information outside ties. As acknowledged in Section 8, larger PRM weights are not tested. I would like to see comparisons with weighted combinations of PRM and rollout scores to clarify whether the conclusions depend on the chosen combination rule.
Questions for authors
  • How reliable is the reference-free approach when the consensus answer is incorrect? Can you report its final-answer accuracy in these cases and whether PRM tie-breaking improves it?

  • How does PRM tie-breaking compare with weighted combinations of PRM and rollout scores on the same candidates and rollout budgets, particularly when only a few rollouts are available?

8 comments
AuthorsOpenReview ↗

Response to Reviewer Gdcy (Part 1/2)

Thank you for your careful review. Your suggestions to separate reference-graded from reference-free conclusions and to test where the consensus answer is wrong made the scope of every claim explicit. We address reliability without reference answers (Q1, W1), why each analysis follows from the previous one (W2) and weighted combinations (Q2, W3). Section, table and appendix numbers refer to the revised PDF, where revisions are in blue.

Q1 / W1. How reliable is the reference-free approach when the consensus answer is wrong, and does PRM tie-breaking improve it?

As reliable as its consensus answer: where that answer is right, consensus value is within 0.15 points of MC value selection, and where it is wrong, final correctness is low and PRM tie-breaking gives no reliable gain. A common wrong answer does mislead consensus value, and the revision measures how much. The revised §1 and §6.3 ("Wrong consensus") say that consensus value is as reliable as its consensus answer. Following your suggestion, we split the selection points by whether the consensus answer, the most common final answer among all branches' source continuations, is right (Appendix E.1, Appendix Table 15):

DataSelection points with a wrong consensus answer (%)Final correctness there (%)Change from PRM tie-breaking there (points)
PRM800K4.6215.10.57 [−0.69, +2.65]
ProcessBench14.899.8−0.50 [−1.25, +0.07]
OpenR148.7610.7−0.32 [−1.52, +0.92]

On OpenR1, your example, consensus value gains 1.40 points [+0.13, +2.74] over uniform selection where the consensus answer is right, and its overall gain is uncertain because about half of its selection points have a wrong consensus answer. The OpenR1 change from PRM tie-breaking covers the 300 selection points whose branches fit Qwen-PRM's context.

W1. Separating reference-graded from reference-free conclusions.

Reference answers are unavailable at test time, so the revision uses them only offline, to price scores on fixed branches and pick a tie-breaker on development problems, and states which conclusions need them (§8). MC value selection reads the reference answer, so it is a diagnostic on fixed branches that measures each score's price (§4 to §6.2). The practical claims rest on the reference-free search (§6.3). At equal generated tokens it does not differ detectably from majority voting on MATH 200 (−0.84 points, [−2.53, +0.75], Appendix B, Table 4), and on repeated selection points of fresh AIME traces at its setting PRM tie-breaking gains 0.16 points per selection point [+0.02, +0.30] (Appendix C.2).

W2. Why each analysis follows from the previous one, and when PRMs are useful.

Each analysis now opens with the question the previous one leaves, and §6 ends with when a PRM is useful: a PRM should be judged by its price at the budget in use, not by its standalone accuracy. §4 is now one paragraph, with the results on each dataset in Appendix A. The chain:

  1. §4 and §5 give the first result, which motivates the price. On ProcessBench, official labels lose as a selector to MC value selection, which grades continuations against the reference answer, yet as a tie-breaker they are worth 1.46 to 1.77 continuations per branch (Table 1).
  2. §6.1 asks what a score knows beyond the continuations on the ties they leave, and proves that its price tends to a weighted average of its preference between close branches (Theorem 2).
  3. §6.2 prices eleven scores. Qwen-PRM and PUM have nearly equal standalone accuracy, yet at four continuations per branch PUM is worth 3.16 continuations and Qwen-PRM 1.05.
  4. §6.3 tests the design under weighted rules, a wrong consensus answer and the complete search.
AuthorsOpenReview ↗

Response to Reviewer Gdcy (Part 2/2)

Q2 / W3. PRM tie-breaking against weighted combinations, on the same branches and budgets.

Under MC value no held-out weighted combination is detectably better than tie-breaking, while under consensus value on ProcessBench a weight helps, most with few continuations and where the consensus answer is wrong. Exact ties are not always the best use of a score, and the revision shows where. Following your suggestion, we ranked branches by X/m+λsX/m+\lambda s on the same branches and budgets, where XX of the mm continuations reach the reference or consensus answer, ss is the score in [0, 1], and λ\lambda is chosen on held-out problems, including one continuation per branch (§6.3, Appendix C.3, Table 8). Weighted rule − tie-breaking, points of final correctness:

Continuations graded againstScores and dataContinuations per branchDifference (points)
Reference answerFive scores, largest difference1 to 80.41 [−0.41, +1.21]
Reference answerQwen-PRM, PRM800K8−0.83 [−1.53, −0.28]
Consensus answerQwen-PRM, ProcessBench12.50 [+1.46, +3.64]
Consensus answerQwen-PRM, ProcessBench4 (the search's budget)0.86 [−0.12, +1.89]
Consensus answerQwen-PRM, ProcessBench8−0.49 [−1.16, +0.15]
Consensus answerQwen-PRM, ProcessBench, consensus answer of all source continuations wrong44.48 [+1.55, +8.19]
Consensus answerQwen-PRM, ProcessBench, consensus answer of all source continuations right40.23 [−0.64, +1.05]

Under consensus value on ProcessBench, a weight helps every score at one continuation per branch, and the official labels and PUM also at four (Appendix C.3). Where the consensus answer is wrong, the counts are biased toward it and a weighted score can outweigh them, while a tie-breaker cannot. Under MC value, a weight that overrides the branch the counts alone rank first can lose accuracy.

We hope the revision answers your concerns, and we would be grateful if you would take these changes into account in your evaluation.

Thank you again for your review. Your main concern was how reliable selection is without reference answers. In the revision, where the consensus answer is right, consensus value comes within 0.15 points of MC value selection, which grades continuations against the reference answer, and on OpenR1, the example you raised, it gains 1.40 points [+0.13, +2.74] over uniform selection (§6.3, Appendix E.1). The weighted rules you proposed are now compared on the same branches and budgets (§6.3, Appendix C.3). Does this address your concern on reliability? If not, which point remains? There is still time before November 18 (AoE) for us to address it.

Reviewer_jNLSOpenReview ↗
Rating
4(+1)
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper asks whether identifying valid reasoning steps is enough to choose the branch that a particular model is most likely to finish correctly. It compares step labels, reward-model scores, and values estimated from sampled continuations on the same candidates, using separate rollouts for selection and evaluation. It then studies using a reward model only to break ties left by sampling. The paper gives a decomposition of selection gains, an identity for the value of one more rollout, and experiments with a reference-free consensus-based search.

Critical strengths
  1. The Introduction's problem is supported by direct comparisons on identical candidates. Official labels remove the reward model's prediction error from the diagnostic, while separate selection and evaluation rollouts reduce the risk of evaluating a selector on the same outcomes that chose its branch. On PRM800K and ProcessBench, the reported differences of 1.71 and 2.65 points have intervals excluding zero. The decomposition and same-label score permutation help explain what the labels miss.

  2. The theoretical arguments are carefully scoped. The extra-rollout identity states its exchangeability and independence assumptions, and Appendix F.4 explicitly explains why it does not directly apply to pooled consensus scoring. The experiments compare the primary score alone, the scorer alone, and their tie-breaking combination, so the main added component is tested. The paper also reports paired intervals for complete search. I found the review format and layout normal, with no obvious overlap or broken equations; a few plot labels are small.

Critical weaknesses
  1. The practical improvement from PRM tie-breaking is less certain than the fixed-candidate diagnosis. In Appendix Table 10, its complete-search voting gains have intervals of -0.67 to 4.50 points on MATH and -1.85 to 10.00 on AIME, both including zero. Some other results decline after adding the scorer, including single-answer MATH and both GPQA measures in Table 4. The 0.16-point gain at individual selection points is positive in the reported interval, but does not establish a reliable improvement for the whole search. Appendix J also makes clear that the intervals hold the generated candidates and rollouts fixed; they do not measure variation from rerunning search. I would therefore treat the problem diagnosis as stronger evidence than the practical remedy. Repeated search runs would help judge whether the tie-breaking module gives a stable benefit.

  2. The reference-graded analysis and the reference-free method should remain clearly separated in the conclusions. The exact rollout-pricing identity concerns reference-graded outcomes, and the matching of consensus and reference scores requires a correct consensus answer. The paper acknowledges this, but those facts alone do not show that the same sampling-budget recommendation holds for the deployed consensus method. Existing consensus budget scans help, yet a clear comparison of the sampling-only and sampling-plus-scorer rules across budgets in complete search would better support the practical recommendation.

Questions for authors
  1. Are independent reruns of the complete search available? In particular, how stable are the MATH and AIME voting gains from adding PRM tie-breaking, whose paired intervals currently include zero?

  2. Do existing complete-search results compare consensus-only selection and consensus with PRM tie-breaking at several rollout budgets? This would help show whether the sampling-budget trend from the reference-graded analysis carries over to the practical method.

  3. Which practical claims do you consider established by the whole-search results, rather than by the fixed-candidate diagnostic or the exact reference-graded identity?

4 comments
AuthorsOpenReview ↗

Response to Reviewer jNLS (Part 1/2)

Thank you for your careful review. Your suggestions of repeated search runs and of a budget comparison under consensus value became two of the revision's main additions. We address the reruns and what the intervals cover (Q1, W1), consensus value across budgets (Q2, W2) and which claims each kind of evidence establishes (Q3). Section, table and appendix numbers refer to the revised PDF, where revisions are in blue.

Q1 / W1. Independent reruns of the complete search, and stability of the MATH and AIME voting gains.

Yes. Over two independent reruns at one continuation per branch, PRM tie-breaking raises Voting detectably, and at the search's budget its contribution is detectable on repeated selection points. A single run per problem leaves wide intervals, so following your suggestion, we reran the complete search on all 290 questions at m=1m=1 (two runs), m=2m=2 (two runs) and m=8m=8 (one run) continuations per branch (Appendix C.2 for the reruns, Appendix Table 5 for the submitted run). PRM tie-breaking − consensus value alone, points:

EvidenceContinuations per branchGain (points)
Two reruns, MATH 200 and AIME 2024–2026, Voting12.18 [+0.07, +4.35]
Submitted run, MATH 200, Voting41.83 [−0.67, +4.50]
Submitted run, AIME 2024–2026, Voting43.89 [−1.85, +10.00]
360 repeated selection points of fresh AIME traces, Single per selection point40.16 [+0.02, +0.30]

Repeated selection points measure the PRM's contribution at the search's settings, one decision at a time, and the complete runs measure it end to end (§6.3).

W1. What the intervals cover.

The intervals include the variation from rerunning the search over problems. The submitted Appendix J said that they hold the generated candidates and continuations fixed, which understated what they cover. Each problem's branches, continuations and search are drawn independently of the other problems, and the bootstrap resamples each problem with its own draws. The revised Appendix D says so.

W1. Results that decline after adding the scorer.

These declines have intervals that include zero or come from an LLM judge rather than a trained PRM. On MATH 200, Single changes by −0.12 points [−3.25, +2.88] (Appendix Table 5). On GPQA Diamond a general LLM judge, Qwen3-14B, breaks the ties (Appendix Table 4). The PRM claim is made for mathematics with a trained PRM.

Q2 / W2. Consensus only against consensus with PRM tie-breaking, at several budgets.

Yes. Under consensus value, the rule the search uses, the PRM's gain falls as the budget grows on the same selection points. Following your suggestion, we compared consensus value with and without PRM tie-breaking at several budgets on the same selection points (Appendix C.1, C.2). PRM gain with fewer continuations minus with more, in points of final correctness on ProcessBench and of Single on AIME:

Selection pointsBranches per pointContinuations per branchDifference (points)
ProcessBench, fixed (322 problems)2 to 41 and 80.97 [0.44, 1.52]
Fresh AIME 2024–2026 traces (360 points)41 and 40.30 [0.15, 0.47]
Fresh AIME 2026 traces (120 points)41 and 40.36 [0.07, 0.65]

The complete-search runs point the same way, with the PRM's gain in Single at m=1m=1 exceeding that at m=8m=8 by 3.19 points [−0.22, +6.59].

W2. Keeping the reference-graded analysis and the reference-free method apart.

The exact identity is proved for MC value; for consensus value, whose consensus answer pools all branches' continuations, the evidence is the paired budget contrasts of Appendix C.1 and C.2. Proposition F.2 proves the identity for continuations graded against the reference answer. Corollary F.4 shows that when agreement and correctness use the same answer equivalence, consensus value gives every branch the same count as MC value wherever the consensus answer is right, so the two select the same branch. §8 states which conclusions need reference answers.

AuthorsOpenReview ↗

Response to Reviewer jNLS (Part 2/2)

Q3. Which practical claims do the whole-search results establish?

The complete search establishes that the method runs end to end without reference answers, while the size of the PRM's contribution and the price rest on repeated selection points and fixed branches. §8 separates the claims that need reference answers from those that do not, and in detail:

EvidenceWhat it establishes
Complete searchRuns end to end without reference answers. At equal generated tokens (Appendix B, Table 4) it does not differ detectably from majority voting on MATH 200 (−0.84 points, [−2.53, +0.75]) and is ahead of last-step RLHFlow-Llama3.1-8B best-of-N there by 3.93 points [+0.96, +7.20]. On AIME 2024–25 its intervals against majority voting, Qwen-PRM best-of-N and last-step RLHFlow best-of-N include zero. PRM tie-breaking raises Voting at m=1m=1 over two reruns by 2.18 points [+0.07, +4.35] (Appendix C.2)
Repeated selection points on fresh AIME tracesThe PRM's gain at the search's setting, 0.16 points per selection point [+0.02, +0.30], and its fall as continuations grow
Fixed branches with reference answersOfficial labels lose to MC value selection (by 1.71 and 2.65 points on PRM800K and ProcessBench), and each score's price in continuations per branch (Table 1)
Exact identityThat the price is exact under MC value (Proposition F.2), and that, under one answer equivalence, consensus value selects as MC value does wherever the consensus answer is right (Corollary F.4)

We hope the revision answers your questions, and we would be grateful if you would take these changes into account in your evaluation.

Thanks for the detailed response, my concerns are fully resolved. I'm increasing the score.

Thank you very much for your reply and for raising your score. Your suggestions to rerun the complete search and to compare budgets under consensus value led to the new results in Appendix C.1 and C.2 and to a clearer statement in §8 of which claims rest on which evidence. We are grateful for your careful review.

Reviewer_iiRgOpenReview ↗
Rating
2(+1)
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This work argues that step validity is a poor proxy for test-time search utility, which depends on the probability of reaching a correct answer from a candidate branch. It compares oracle step-validity labels against Monte Carlo rollout-based value estimates across reasoning benchmarks.

The main findings are that (a) MC value with 4–8 rollouts outperforms oracle validity labels by 1.71–2.65 percentage points, (b) PRMs used as tie-breakers are worth approximately one additional rollout at small budgets, and (c) this advantage disappears as rollout budgets increase. A multi-round search experiment finds only a 0.16-point improvement from PRM tie-breaking.

Critical strengths
  1. Proposition F.2 (one more rollout equals its expected tie-resolution gain) provides an exact way to express the benefit of PRM tie-breaking in additional rollouts (although the measured benefit is small)
  2. The approach used in separating selection and evaluation rollouts to avoid selection bias is great. Using oracle validity labels isolates the mismatch between validity and recoverability from PRM prediction errors. However, it does not offset the limited novelty and scope in the weaknesses listed below.
Critical weaknesses
  1. Distinction between step validity and eventual success is already well established. Setlur et al. [R1] and Zhou et al. [R2] discuss this mismatch directly. Outcome value models (Yu et al. [R3]; Wang et al. [R4]) and Math-Shepherd (Wang et al. [R5]) already use rollout-based success signals. Cinquin et al. [R6] report limited benefits from PRM-guided search.

    So, the theoretical contributions are largely elementary accounting identities. Proposition 1 decomposes gains into between-group and within-group terms, and inability of binary labels to rank candidates within the same class is trivial. The paper provides limited new conceptual or methodological insight.

  2. Shown analysis obly focuses on PRMs used only as lexicographic tie-breakers, rather than standard PRM-guided search or weighted scoring. And the observed improvements are small, e.g., 0.16 points per selection point in full search, with confidence intervals spanning zero. Tie-breaking gains approach zero by eight rollouts. The strongest MC-value experiments require reference answers, making the selector impractical at inference time.

  3. Table 4 compares consensus-value search using 16 rollouts per round per selection point against substantially cheaper direct-generation and PRM-guided baselines, so it doesn't really seem like a fair comparison. Without matching compute in some way, results cannot establish whether the gains come from the selection method or simply additional sampling.

References.

  • [R1] Setlur, Amrith, et al. "Rewarding progress: Scaling automated process verifiers for llm reasoning." International Conference on Learning Representations. Vol. 2025. 2025.
  • [R2] Zhou, Yuhang, Yixin Cao, and Guangnan Ye. "From Correctness to Utility: Gain-Based Prefix Evaluation for LLM Reasoning." arXiv preprint arXiv:2606.07190 (2026).
  • [R3] Yu, Fei, Anningzhe Gao, and Benyou Wang. "Ovm, outcome-supervised value models for planning in mathematical reasoning." Findings of the Association for Computational Linguistics: NAACL 2024. 2024.
  • [R4] Wang, Kaiwen, et al. "Value-guided search for efficient chain-of-thought reasoning." Advances in Neural Information Processing Systems 38 (2026): 84806-84841.
  • [R5] Wang, Peiyi, et al. "Math-shepherd: Verify and reinforce llms step-by-step without human annotations." Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). 2024.
  • [R6] Cinquin, Tristan, Geoff Pleiss, and Agustinus Kristiadi. "Limits of PRM-Guided Tree Search for Mathematical Reasoning with LLMs." arXiv preprint arXiv:2510.20272 (2025).
Questions for authors
  1. What is the main novel result beyond prior work establishing the mismatch between validity and success probability?
  2. Can the authors provide compute-matched comparisons against majority voting and PRM best-of-N, using the same total generated-token budget as consensus-value search?
  3. How sensitive are the fixed-candidate results to text-reformatting prefixes, given that the full-search experiments resume original tokens?
  4. Why restrict the analysis to lexicographic tie-breaking rather than more practical combinations such as weighted PRM scores or PRM-weighted voting?
7 comments
AuthorsOpenReview ↗

Response to Reviewer iiRg (Part 1/2)

Thank you for your careful review. Your question of what is new beyond the known mismatch between validity and success led us to put the price of a score at the centre. We address novelty (Q1, W1), format sensitivity (Q3), weighted rules and gain size (Q4, W2) and compute-matched baselines (Q2, W3). Section, table and appendix numbers refer to the revised PDF, where revisions are in blue.

Q1 / W1. What is new beyond the known mismatch between validity and success?

Two things are new: an exact price of a score in sampled continuations (Proposition F.2, in the submission) and a theorem on what sets this price (Theorem 2, added in the revision).

  • Known. Validity and success differ [R1, R2], which the revised §1 cites as background, but prior work leaves open what a score is worth once sampled continuations score the branches.
  • New. The price makes this worth exact: it is the number of extra continuations per branch that gain as much as the score does on the ties the continuations leave (Proposition F.2). As continuations grow, the price provably tends to a weighted average of the score's preference between close branches, not to its standalone accuracy (Theorem 2).
  • Evidence. Qwen-PRM (78.6) and PUM (78.2) have nearly equal standalone accuracy but differ threefold in price. At eight continuations per branch, this preference orders the eleven scores' prices with Spearman 0.90 and 0.93; standalone accuracy and existing measures reach at most 0.68 (Appendix Table 11).

Price of each score in extra continuations per branch, on ProcessBench with continuations graded against the reference answer (§6.2, Table 1):

Score1 continuation per branch24
Official labels1.461.721.77
Qwen-PRM0.911.061.05
Skywork-PRM (1.5B)0.861.492.38
PUM [R2]1.442.503.16
VGS value model [R4]0.761.662.70

A score should therefore be chosen by its price at the budget in use, not by its standalone accuracy. As continuations grow, its preference between close branches sets this price.

W1. The theoretical contributions as elementary identities.

Propositions A.1 and F.2 are exact tools, and the theory of the revision is Theorem 2 and Proposition F.8 (§6.1, Appendix F). Proposition A.1 (Proposition 1 of the submission) splits a selector's gain into a part between label groups and a part within them, and Proposition F.2 lets the price be read off recorded continuations. As continuations grow, ties remain only between branches of close recoverability, so only the score's ranking of such branches sets the price (Theorem 2). Proposition F.8 gives conditions under which, with two branches, the ratio of the score's gain to the gain of one more continuation does not fall as continuations grow.

W1. Relation to [R1]–[R6].

Each line of work leaves open a question that the price answers. The revised §1 answers each in a table, and Appendix C.4 answers [R5] (§1, Appendix C.4):

Prior workOur answer
[R1] Alignment and distinguishability of proversQwen-PRM and PUM are nearly equal in analogues of both, yet differ threefold in price (Appendix C.4)
[R2] PUM, a solve-rate gainWorth 3.16 continuations at four per branch on ProcessBench (Table 1)
[R3] An outcome value model guides decodingWhat a value model adds on the ties that continuations leave, priced for the VGS value model (§6.2)
[R4] A token-level value model guides searchThe VGS value model's gain in points falls as continuations grow, yet its price rises from 0.76 to 2.70 continuations (Table 1, Appendix Table 6)
[R5] Labels from continuation successAt four continuations per branch on ProcessBench, the fraction label gains 3.59 times as much as one more continuation, the any-success label about 0.3 times (Appendix Table 12)
[R6] Limits of PRM-guided searchThe score's preference between close branches sets its worth (Theorem 2)

Q3. Sensitivity of the fixed-branch results to reformatted prefixes.

On all 1,785 PRM800K branches, the search's own format changes MC values about as much as a second draw in the original format does. Following your suggestion, we continued these branches in the search's own format (Appendix E.2):

ComparisonValue
Correlation of MC values, search format with original0.81
Correlation of MC values, second original draw with original0.83
Second original draw − search format0.02 [0.00, +0.05]
Change in the label-satisfaction gap between validity selection and MC value selection (points)0.42 [−0.82, +1.75]
AuthorsOpenReview ↗

Response to Reviewer iiRg (Part 2/2)

Q1. A test of the choice the price implies, on new questions.

The PRM picked by its preference between close branches has the larger point estimate in all three settings, and leads detectably under consensus value. Before scoring any new branch, we picked a tie-breaker by each rule on 90 development AIME questions: the preference rule picked EurusPRM-Stage2, and standalone accuracy picked Qwen-PRM. We then tested both on 225 AIME questions never used in this study (Appendix C.5, Table 13). EurusPRM-Stage2 − Qwen-PRM, points of Single per selection point:

Continuations graded againstContinuations per branchDifference (points)In units of one more continuation
Reference answer8+0.164 [−0.009, +0.340]+1.55 [−0.09, +3.25]
Reference answer16+0.090 [−0.026, +0.206]+1.58 [−0.45, +3.74]
Most common answer (consensus value)8+0.182 [+0.013, +0.353]+2.10 [+0.15, +4.20]

Q4 / W2. Why lexicographic tie-breaking rather than weighted PRM scores or PRM-weighted voting?

Tie-breaking is how we measure what a score knows beyond the continuations, and under MC value no held-out weighted rule beats it detectably. The revised §1 and §6.3 call it a measurement design. On a tie the counts do not separate the branches, so every change that tie-breaking makes comes from the score. Following your suggestion, we ranked branches by X/m+λsX/m+\lambda s on the same branches and budgets, where XX of the mm continuations reach the reference or consensus answer, ss is the score in [0, 1], and λ\lambda is chosen on held-out problems (§6.3, Appendix C.3). Weighted rule − tie-breaking, points of final correctness:

Continuations graded againstScore, data and continuations per branchDifference (points)
Reference answerFive scores, 1 to 8, largest difference0.41 [−0.41, +1.21]
Consensus answerQwen-PRM, ProcessBench, 12.50 [+1.46, +3.64]
Consensus answerQwen-PRM, ProcessBench, 4 (the search's budget)0.86 [−0.12, +1.89]

Under consensus value on ProcessBench, a weight helps, most with few continuations and where the consensus answer is wrong, since there the counts are biased toward that answer. At the search's generated tokens on MATH 200, PRM-weighted voting reaches 95.38%, majority voting 95.34% and the search 94.50% (Appendix Table 4).

W2. The size of the gains.

On ProcessBench, PUM is worth 3.16 continuations per branch at four per branch, and the prices of three scores rise with the budget (§6.2, Table 1). One more continuation also gains few points, since it too acts only on the current top ties (Proposition F.2). The price therefore compares the score with one more continuation on the same selection points. The 0.16 points per selection point come from repeated selection points of fresh AIME traces, not from the complete search, and their interval [+0.02, +0.30] excludes zero (Appendix C.2).

W2. Reliance on reference answers.

MC value is a diagnostic on fixed branches, and the search uses consensus value, which needs no reference answer (§6.3). §8 states which conclusions need reference answers.

Q2 / W3. Compute-matched comparison with majority voting and PRM best-of-N.

At equal generated tokens, the search does not differ detectably from majority voting and is ahead of RLHFlow best-of-N on MATH 200. The submitted baselines used fewer generated tokens. Following your suggestion, each baseline now samples as many direct solutions as fit in the search's generated tokens, 59 to 60 on MATH 200 and 63 to 64 on AIME 2024–25 (Appendix Table 4, intervals in Appendix B). The budget counts generated policy tokens, and PRM reads are counted apart, in policy-token equivalents. Voting (%) and search − baseline (points):

At the search's generated tokensMATH 200Search − baselineAIME 2024–25Search − baseline
The search94.5060.28
Majority voting95.34−0.84 [−2.53, +0.75]66.51−6.23 [−14.40, +1.39]
Qwen-PRM best-of-N92.53+1.97 [−0.49, +4.66]58.33+1.94 [−7.78, +11.39]
RLHFlow-Llama3.1-8B best-of-N (last step)90.57+3.93 [+0.96, +7.20]54.97+5.31 [−3.53, +14.42]

Each baseline generates as many policy tokens as the search, the budget you asked for. In policy-token equivalents, the PRMs of best-of-N also read 2.0 to 2.6 times these tokens, while the search's Qwen-PRM reads 4.7% of them.

We hope the revision answers your questions, and we would be grateful if you would take these changes into account in your evaluation.

Thanks a lot for providing a detailed response. It has addressed some of my concerns and accordingly, I have raised my score.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Summary of the final revision

We thank the reviewers and the AC for their time. Reviewers noted that Proposition F.2 prices PRM tie-breaking exactly in extra continuations (iiRg), that separate selection and evaluation continuations avoid selection bias (iiRg, jNLS), that the theoretical arguments are carefully scoped (jNLS) and that the paper highlights recoverability as an important consideration in branch selection (Gdcy). After our replies, Reviewer jNLS wrote that the concerns are fully resolved and raised the score, and Reviewer iiRg raised the score. This post lists the changes in the revised PDF, marked in blue, by question. The price of a score is the number of extra sampled continuations per branch that gain as much as the score does on the ties the continuations leave.

Question (raised by)What we addedWhere
What is new (iiRg)Theorem 2 shows that, under its assumptions, as sampling grows a score's price follows its preference between close branches, not its standalone accuracy. Qwen-PRM and PUM have nearly the same standalone accuracy, yet at four continuations per branch PUM is worth 3.16 continuations and Qwen-PRM 1.05. On 225 new AIME questions, the score this preference picks beats the pick of standalone accuracy under consensus value by +0.18 [+0.01, +0.35] points per selection point§1, §6, App. C.5
Compute fairness (iiRg)At about equal total cost, with PRM reads charged, the search's vote over all its answers is ahead of majority voting on AIME 2024–25 by +5.83 [+0.42, +12.50] at four continuations per branch. At equal total cost the search's four-answer vote is also ahead of RLHFlow best-of-N (last step) on both benchmarks at one, two and four continuations per branch and of Qwen-PRM best-of-N on MATH at one and two, with intervals above zeroTable 4, App. C.2
ORM best-of-N (Gdcy)With the same vote (post hoc), the search is ahead of ORM best-of-N at equal total cost on AIME 2024–25 at two and four continuations per branch, by up to +8.83 [+3.25, +14.64]App. C.2
Weighted rules (iiRg, Gdcy)No held-out weighted rule is detectably better than tie-breaking under MC value, nor under consensus value where at least 80% of the sampled answers agree, on four datasets at four continuations per branch§6.3, App. C.3
Price under the deployed rule (jNLS, iiRg)Prices of eleven scores on ProcessBench measured under consensus value. App. F.3 keeps the bound of Corollary F.4 on the gap between consensus value and MC valueTable 10, App. F.3
Size of the gains (iiRg)With four branches and one continuation each, a Qwen-PRM read saves 1.06 [0.21, 3.14] continuations per branch, about half of what uniform ties needApp. C.2
Prefix format (iiRg)The search's own format changes the results no more than a second bank in the original format doesApp. E.2
Reliability without reference answers (Gdcy)Where at least 80% of the sampled answers agree, a share known at test time, the deployed rule shows no detectable gap to MC value selection on all four datasets at four continuations per branch§6.3, App. E.1
Use without reference answers (Gdcy, jNLS)The deployed rule never reads them. §8 and a new table in App. B.1 state which results rest on the complete search and which on fixed branches, and that reference answers serve only offline, to price scores and pick a tie-breaker, and otherwise only grade the final answer. In a post hoc replay where both methods stop once 80% of their first answers agree, at about equal mean cost, the search leads majority voting on AIME 2024–25 by +6.14 [+0.72, +12.50] at four continuations per branch§8, App. B.1, C.2
Structure (Gdcy)§4 to §6 rewritten as one chain of findings. §4 and §5 give the first result, official labels lose as a selector to MC value selection yet are worth 1.46 to 1.77 continuations per branch on ProcessBench. §6 asks what sets the price, prices eleven scores and tests the design§4 to §6
Stability over reruns (jNLS)Over two reruns at one continuation per branch, PRM tie-breaking raises Voting by 2.18 [+0.07, +4.35]App. C.2
Budget trend (jNLS)On the same selection points, the PRM's gain under consensus value is also smaller at the larger budgetApp. C.1

Appendices B, F and G add a reproduction of Table 6 of Zhang et al. (2025), the formal Theorem 2 and a notation table, and cited items are renumbered as follows.

Cited in the reviewsIn the revision
Section 4.1§4
Proposition 1Proposition A.1
Table 4Table 4, in App. B
Table 10Table 5
Table 18Table 15
Appendix F.4App. F.3

We thank the reviewers again for comments that improved the paper.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.