- Rating
- 3
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper proposes BetaPRM, a process reward model that predicts both the success probability of a reasoning prefix and the reliability of that prediction. It uses a Beta-Binomial objective to learn from Monte Carlo continuation counts. The authors further introduce Adaptive Computation Allocation (ACA), which uses the learned uncertainty to guide early stopping and prefix expansion. Experiments across four model backbones and four reasoning benchmarks show improved Best-of-N selection accuracy, comparable step-level error detection, and up to 33.57% reduction in generated tokens with ACA.
Critical strengths
- The paper is well written and clearly organized, with intuitive figures that effectively illustrate the motivation and key ideas. The proposed BetaPRM provides a principled way to model both process rewards and their reliability.
- Extensive experiments across four backbones and four benchmarks demonstrate consistent improvements in Best-of-N selection. The ablation studies further validate the effectiveness of the learned reliability signal.
- The proposed ACA effectively uses uncertainty to guide computation allocation, reducing token consumption by up to 33.57% while improving final-answer accuracy.
Critical weaknesses
- The motivation for Beta-Binomial supervision is not fully justified, as cross-entropy with Monte Carlo success ratios remains statistically valid despite sampling noise. Moreover, the calibration of the learned reliability estimates is not sufficiently established.
- ACA is only evaluated under a Best-of-16 budget. Its effectiveness under larger budgets (e.g., Best-of-32/64) remains unclear, and compute-matched comparisons are lacking.
- The paper lacks sufficient discussion and comparison with related work such as AdaptiveStep, which also leverages confidence signals to identify important reasoning steps and guide generation. This makes it difficult to assess the unique benefits of the proposed uncertainty-guided prefix repair strategy.
Questions for authors
- Could the authors further clarify the advantages of Beta-Binomial supervision over standard cross-entropy and provide additional evidence of uncertainty calibration?
- Could the authors evaluate ACA under larger budgets (e.g., Best-of-32/64) and provide compute-matched comparisons to better demonstrate its scalability and efficiency?
- Could the authors discuss the relationship with AdaptiveStep and consider comparing against a confidence-based step selection strategy under the same computational budget?