acceptodds
Under review as a conference paper at ICLR 2027

Process Rewards with Learned Reliability

Abstract

Process Reward Models (PRMs) provide step-level feedback for reasoning, but current PRMs usually output only a single reward score for each step. Downstream methods must therefore treat imperfect step-level reward predictions as reliable decision signals, with no indication of when these predictions should be trusted. We propose BetaPRM, a distributional PRM that predicts both a step-level success probability and the reliability of that prediction. Given step-success supervision from Monte Carlo continuations, BetaPRM learns a Beta belief that explains the observed number of successful continuations through a Beta-Binomial likelihood, rather than regressing to the finite-sample success ratio as a point target. This learned reliability signal indicates when a step reward should be trusted, enabling downstream applications to distinguish reliable rewards from uncertain ones. As one application, we introduce Adaptive Computation Allocation (ACA) for PRM-guided Best-of-N reasoning. ACA uses the learned reliability signal to stop when a high-reward solution is reliable and to spend additional computation on uncertain candidate prefixes. Experiments across four backbones and four reasoning benchmarks show that BetaPRM improves PRM-guided Best-of-N selection while preserving standard step-level error detection. Built on this signal, ACA improves the accuracy–token tradeoff over fixed-budget Best-of-16, reducing token usage by up to 33.57% while improving final-answer accuracy.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Process Reward Model, Test-Time Scaling

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_uYiROpenReview ↗
Rating
3
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes BetaPRM, a process reward model that predicts both the success probability of a reasoning prefix and the reliability of that prediction. It uses a Beta-Binomial objective to learn from Monte Carlo continuation counts. The authors further introduce Adaptive Computation Allocation (ACA), which uses the learned uncertainty to guide early stopping and prefix expansion. Experiments across four model backbones and four reasoning benchmarks show improved Best-of-N selection accuracy, comparable step-level error detection, and up to 33.57% reduction in generated tokens with ACA.

Critical strengths
  1. The paper is well written and clearly organized, with intuitive figures that effectively illustrate the motivation and key ideas. The proposed BetaPRM provides a principled way to model both process rewards and their reliability.
  2. Extensive experiments across four backbones and four benchmarks demonstrate consistent improvements in Best-of-N selection. The ablation studies further validate the effectiveness of the learned reliability signal.
  3. The proposed ACA effectively uses uncertainty to guide computation allocation, reducing token consumption by up to 33.57% while improving final-answer accuracy.
Critical weaknesses
  1. The motivation for Beta-Binomial supervision is not fully justified, as cross-entropy with Monte Carlo success ratios remains statistically valid despite sampling noise. Moreover, the calibration of the learned reliability estimates is not sufficiently established.
  2. ACA is only evaluated under a Best-of-16 budget. Its effectiveness under larger budgets (e.g., Best-of-32/64) remains unclear, and compute-matched comparisons are lacking.
  3. The paper lacks sufficient discussion and comparison with related work such as AdaptiveStep, which also leverages confidence signals to identify important reasoning steps and guide generation. This makes it difficult to assess the unique benefits of the proposed uncertainty-guided prefix repair strategy.
Questions for authors
  1. Could the authors further clarify the advantages of Beta-Binomial supervision over standard cross-entropy and provide additional evidence of uncertainty calibration?
  2. Could the authors evaluate ACA under larger budgets (e.g., Best-of-32/64) and provide compute-matched comparisons to better demonstrate its scalability and efficiency?
  3. Could the authors discuss the relationship with AdaptiveStep and consider comparing against a confidence-based step selection strategy under the same computational budget?
Reviewer_zogTOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes BetaPRM, a process reward model that predicts both reward means and uncertainty for intermediate reasoning steps via a learned concentration parameter. Experiments on visual mathematical reasoning show improved step-level error detection over base models. Best-of-16 evaluations further demonstrate that BetaPRM improves final-answer accuracy over Standard PRM, with additional gains from uncertainty-aware selection.

Critical strengths

BetaPRM supports predicting the success probability along with the uncertainty, which can be used in the downstream Best-of-N searching to accept the reliable answer without extra sampling via early stopping.

Critical weaknesses
  • Although ACA substantially reduces token consumption, its improvements in final-answer accuracy appear relatively modest, with gains of approximately 1–2 percentage points over Vanilla BoN across the evaluated benchmarks. Further analysis would help clarify the conditions under which ACA delivers meaningful accuracy improvements in addition to token savings.

  • The experimental evaluation is limited to vanilla baselines and lacks comparisons with other state-of-the-art PRMs. Including such comparisons would strengthen the empirical evaluation and provide a clearer assessment of the effectiveness of proposed methods relative to existing approaches.

Questions for authors
  • Table 5 shows that ACA without early stopping reduces token consumption but generally does not improve accuracy over Vanilla BoN, whereas the full ACA consistently improves accuracy across all four benchmarks and both backbones. Could the authors provide further analysis to substantiate the hypothesis that generating additional candidates after a sufficiently reliable answer has been found can mislead an imperfect PRM selector?

  • Given that all evaluations are conducted on mathematical reasoning benchmarks, could the authors clarify why they do not use a math-specialized model as the backbone for the PRM? Evaluating this setting would help assess whether BetaPRM benefits extend to PRMs built on models with stronger mathematical reasoning capabilities.

Reviewer_5c6XOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper focuses on guiding large language models' mathematical reasoning with rewards coming from a process reward model. The paper tackles an important known shortcoming of such models, which is the lack of reliability in the observed rewards for partial answers that carry noise due to the nature of Monte Carlo sampling with finite samples. The paper suggests fitting a distribution, specifically a beta distribution, to the observed success rate of partially generated answers to estimate both the reward, the mean of the distribution, and reliability, the concentration of the distribution. The authors then suggest an adaptive allocation strategy that utilizes the distributional fit to propose a better alternative to best-of-n sampling.

Critical strengths

The paper focuses on a fundamental issue of process reward models, which is the noisiness of the rewards that come from the current way of training such models. To address this, the authors suggest a distributional fit instead of regressing to the mean, an interesting direction to explore further.

Critical weaknesses

Justifying and motivating the methodology chosen for fitting can be improved, in my opinion. There are some design choices that are not well justified in the paper. examples: "κt = softplus(gϕ(ht)) + κmin", what is softplus? Why is this natural? "Because qt is an unknown latent success probability in [0, 1], we model it with a Beta belief, qt ∼ Beta(αt, βt), which naturally pairs with the Binomial count observation above." What is the logical argument behind this sentence and modeling?

There are various alternatives to best-of-N that have been proposed and have shown promising results, including [1], which is an important missing citation; the paper lacks any sort of comparison with these methods. Further, the idea of adaptively allocating the compute based on PRM scores has been proposed before, including [2], which paper lack comparisons with this methods as well.

[1] Puri, Isha, et al. "Rollout roulette: A probabilistic inference approach to inference-time scaling of llms using particle-based monte carlo methods." Advances in Neural Information Processing Systems 38 (2026)

[2] Park, Young-Jin, et al. "Know What You Don't Know: Uncertainty Calibration of Process Reward Models." Advances in neural information processing systems 38 (2026)

Questions for authors

Related to the weakness section

  1. Could you please justify fitting a beta distribution to the observed empirical rewards? I
  2. Would it be possible to compare against other alternatives that have been proposed for best-of-N and adaptive computing schemes for PRM-guided mathematical reasoning? To clarify, I'm not asking the method to outperform all the previous algorithms, but I would expect to see a comparison with other methods in the paper. Tasks such as MATH500 and AIME are well studied and can be a good benchmark to compare.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.