- Rating
- 1
- Confidence
- 2
- Informativeness
- 1
- Presentation
- 1
- Soundness
- 1
Read review
Paper summary
This paper studies chain-of-thought branch selection, where process reward models (PRMs) are commonly used to favor valid reasoning steps. The authors argue that selection should also consider how promising each branch is for reaching a correct final answer. They therefore consider bounded recoverability, the probability that a given model completes a branch correctly within a specified token budget.
The authors compare selection using reference-graded continuations, which are checked against known answers, with reference-free selection based on agreement among sampled answers. Their experiments show that step validity alone is insufficient for identifying promising branches. They further use PRMs to break ties between branches with equal continuation-based scores. The results suggest that PRMs can provide useful additional guidance in this role, particularly when only a few continuations are available, while their contribution generally decreases as sampling increases.
Critical strengths
- The paper highlights recoverability as an important consideration in branch selection, showing that a model can still reach a correct final answer even when an intermediate reasoning step is invalid.
Critical weaknesses
- The practical applicability of the main MC-based selection analysis is limited by its reliance on reference answers. Although acknowledged in Section 8, this is a substantial limitation because correct answers are generally unavailable when solving new problems at test time. Gold-answer verification supplies information directly aligned with the final-correctness metric, so the resulting advantage over validity-based selection does not by itself establish a practical advantage over PRMs. The authors also evaluate a reference-free consensus variant, but agreement among sampled answers can favor a common wrong answer. For example, on OpenR1, the consensus answer is frequently incorrect, and the resulting accuracy improvement over uniform selection remains uncertain (Table 18). The paper should more clearly distinguish the conclusions supported by reference-graded experiments from those established without reference answers, and provide stronger evidence for the reliability of the reference-free approach.
- I found parts of the results section difficult to follow. Section 4.1 moves quickly between results on different datasets and tasks, with little explanation of what they establish together. Section 6 similarly moves from tie-breaking experiments to comparisons with extra rollouts and then to sequential search. The paper provides explanations and takeaways, but the many numerical comparisons make the main argument harder to follow. I would like the authors to explain more clearly why each analysis follows from the previous one and discuss what the results collectively tell us about when PRMs are useful.
- The study of combining PRM scores with rollout evidence focuses on tie-breaking. This isolates the PRM's contribution, but does not establish whether restricting it to exact ties is the most effective use of its scores. With only a few rollouts, small differences between sampled scores may be unreliable, and PRM scores could still provide useful information outside ties. As acknowledged in Section 8, larger PRM weights are not tested. I would like to see comparisons with weighted combinations of PRM and rollout scores to clarify whether the conclusions depend on the chosen combination rule.
Questions for authors
-
How reliable is the reference-free approach when the consensus answer is incorrect? Can you report its final-answer accuracy in these cases and whether PRM tie-breaking improves it?
-
How does PRM tie-breaking compare with weighted combinations of PRM and rollout scores on the same candidates and rollout budgets, particularly when only a few rollouts are available?
8 comments
Response to Reviewer Gdcy (Part 2/2)
Q2 / W3. PRM tie-breaking against weighted combinations, on the same branches and budgets.
Under MC value no held-out weighted combination is detectably better than tie-breaking, while under consensus value on ProcessBench a weight helps, most with few continuations and where the consensus answer is wrong. Exact ties are not always the best use of a score, and the revision shows where. Following your suggestion, we ranked branches by on the same branches and budgets, where of the continuations reach the reference or consensus answer, is the score in [0, 1], and is chosen on held-out problems, including one continuation per branch (§6.3, Appendix C.3, Table 8). Weighted rule − tie-breaking, points of final correctness:
| Continuations graded against | Scores and data | Continuations per branch | Difference (points) |
|---|---|---|---|
| Reference answer | Five scores, largest difference | 1 to 8 | 0.41 [−0.41, +1.21] |
| Reference answer | Qwen-PRM, PRM800K | 8 | −0.83 [−1.53, −0.28] |
| Consensus answer | Qwen-PRM, ProcessBench | 1 | 2.50 [+1.46, +3.64] |
| Consensus answer | Qwen-PRM, ProcessBench | 4 (the search's budget) | 0.86 [−0.12, +1.89] |
| Consensus answer | Qwen-PRM, ProcessBench | 8 | −0.49 [−1.16, +0.15] |
| Consensus answer | Qwen-PRM, ProcessBench, consensus answer of all source continuations wrong | 4 | 4.48 [+1.55, +8.19] |
| Consensus answer | Qwen-PRM, ProcessBench, consensus answer of all source continuations right | 4 | 0.23 [−0.64, +1.05] |
Under consensus value on ProcessBench, a weight helps every score at one continuation per branch, and the official labels and PUM also at four (Appendix C.3). Where the consensus answer is wrong, the counts are biased toward it and a weighted score can outweigh them, while a tie-breaker cannot. Under MC value, a weight that overrides the branch the counts alone rank first can lose accuracy.
We hope the revision answers your concerns, and we would be grateful if you would take these changes into account in your evaluation.
Thank you again for your review. Your main concern was how reliable selection is without reference answers. In the revision, where the consensus answer is right, consensus value comes within 0.15 points of MC value selection, which grades continuations against the reference answer, and on OpenR1, the example you raised, it gains 1.40 points [+0.13, +2.74] over uniform selection (§6.3, Appendix E.1). The weighted rules you proposed are now compared on the same branches and budgets (§6.3, Appendix C.3). Does this address your concern on reliability? If not, which point remains? There is still time before November 18 (AoE) for us to address it.
Response to Reviewer Gdcy (Part 1/2)
Thank you for your careful review. Your suggestions to separate reference-graded from reference-free conclusions and to test where the consensus answer is wrong made the scope of every claim explicit. We address reliability without reference answers (Q1, W1), why each analysis follows from the previous one (W2) and weighted combinations (Q2, W3). Section, table and appendix numbers refer to the revised PDF, where revisions are in blue.
Q1 / W1. How reliable is the reference-free approach when the consensus answer is wrong, and does PRM tie-breaking improve it?
As reliable as its consensus answer: where that answer is right, consensus value is within 0.15 points of MC value selection, and where it is wrong, final correctness is low and PRM tie-breaking gives no reliable gain. A common wrong answer does mislead consensus value, and the revision measures how much. The revised §1 and §6.3 ("Wrong consensus") say that consensus value is as reliable as its consensus answer. Following your suggestion, we split the selection points by whether the consensus answer, the most common final answer among all branches' source continuations, is right (Appendix E.1, Appendix Table 15):
On OpenR1, your example, consensus value gains 1.40 points [+0.13, +2.74] over uniform selection where the consensus answer is right, and its overall gain is uncertain because about half of its selection points have a wrong consensus answer. The OpenR1 change from PRM tie-breaking covers the 300 selection points whose branches fit Qwen-PRM's context.
W1. Separating reference-graded from reference-free conclusions.
Reference answers are unavailable at test time, so the revision uses them only offline, to price scores on fixed branches and pick a tie-breaker on development problems, and states which conclusions need them (§8). MC value selection reads the reference answer, so it is a diagnostic on fixed branches that measures each score's price (§4 to §6.2). The practical claims rest on the reference-free search (§6.3). At equal generated tokens it does not differ detectably from majority voting on MATH 200 (−0.84 points, [−2.53, +0.75], Appendix B, Table 4), and on repeated selection points of fresh AIME traces at its setting PRM tie-breaking gains 0.16 points per selection point [+0.02, +0.30] (Appendix C.2).
W2. Why each analysis follows from the previous one, and when PRMs are useful.
Each analysis now opens with the question the previous one leaves, and §6 ends with when a PRM is useful: a PRM should be judged by its price at the budget in use, not by its standalone accuracy. §4 is now one paragraph, with the results on each dataset in Appendix A. The chain: