acceptodds
Under review as a conference paper at ICLR 2027

Unsupervised Process Reward Models

Abstract

Process Reward Models (PRMs) are a powerful mechanism for steering large language model reasoning by providing fine-grained, step-level supervision. However, this effectiveness comes at a significant cost: PRMs require expert annotations for every reasoning step, making them costly and difficult to scale. Here, we propose a method for training unsupervised PRMs (uPRM) that requires no human supervision, neither at the level of step-by-step annotations nor through ground-truth verification of final answers. The key idea behind our approach is to define a scoring function, derived from LLM next-token probabilities, that jointly assesses candidate positions of first erroneous steps across a batch of reasoning trajectories. We demonstrate the effectiveness of uPRM across diverse scenarios: (i) uPRM achieves up to 15% absolute accuracy improvements over the LLM-as-a-Judge in identifying first erroneous steps on the ProcessBench dataset; (ii) as a verifier for test-time scaling, uPRM performs comparably to supervised PRMs and outperforms the majority voting baseline by up to 7%, and (iii) when used as a reward signal in reinforcement learning, uPRM enables more robust policy optimization throughout training compared to a supervised PRM trained using ground-truth labels. Overall, our results open a path toward scalable reward modeling for complex reasoning tasks.

Primary area:
foundation or frontier models, including LLMs
Keywords:
process reward models, reasoning, large language models, unsupervised learning

Reviews

Among:
Shape:
1234this paper, Bottom 9%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.50 over 2 reviews, above 9% of the others.
Reviewer_69xuOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

The paper introduces Unsupervised Process Reward Models (uPRMs). For each reasoning trajectory, the method samples a candidate first-error position and inserts correctness markers into the trajectory. A frozen LLM scores these assignments through the conditional probabilities of the markers, jointly processing multiple trajectories so that earlier examples provide context for subsequent judgments. The PRM is trained to maximize this score using entropy-regularized reinforcement learning, with an additional correction discouraging degenerate label assignments. Training uses reasoning trajectories from PRM800K without their step or outcome labels.

Critical strengths

The method removes both forms of task-specific correctness supervision during PRM training, drawing its signal from a frozen LLM’s marker probabilities. This is consequential because it offers a practical route to building process verifiers where reliable annotations or answer checkers are unavailable.

Critical weaknesses
  1. The contribution of joint scoring is insufficiently isolated. A controlled comparison with independent scoring under otherwise identical training settings is missing.

  2. Evidence for resistance to reward hacking remains limited. uPRM still exhibits hacking in some settings, and sensitivity to reward calibration and PURE hyperparameters remains unexplored. Reporting the last non-degenerate checkpoints also complicates comparisons under equal training budgets.

  3. The term “unsupervised” may be inappropriate, as the supervision signal comes from an external model.

Questions for authors

PRM800K appears outdated and insufficiently challenging.

2 comments
AuthorsOpenReview ↗

Response to Reviewer 69xu

The contribution of joint scoring is insufficiently isolated. A controlled comparison with independent scoring under otherwise identical training settings is missing.

We thank the reviewer for their comment. We would like to highlight that this comparison is identical to the LLM-as-a-Judge (Table 1), because the judge is the optimum of our training objective under independent scoring.

With independent scoring (N=1N=1), Eq. (12) decouples across trajectories into
maxpEj∼p[S(j;s)]+γH(p)max_{p}\mathbb{E}_{j \sim p}[S(j;s)] + \gamma H(p)

whose maximizer is pγ∗(j∣τ)=exp(S(j;s)/γ)∑kexp(S(k;s)/γ)p_{\gamma}^{*}(j \vert \tau) = \frac{\text{exp}(S(j;s)/\gamma)}{\sum_{k}\text{exp}(S(k;s)/\gamma)}

By Eq. (6), qLLM(j∣τ)=exp(S(j;s))q_{LLM}(j \vert \tau) = \text{exp}(S(j;s)).

Thus, the maximizer above is just temperature-adjusted qLLM(j∣τ)1/γq_{LLM}(j \vert \tau)^{1/\gamma}, whose argmax is exactly the naive judge from Table 1 for any temperature γ\gamma. The gains brought by uPRM (Table 1) are therefore attributable to joint scoring.

Evidence for resistance to reward hacking remains limited. uPRM still exhibits ... under equal training budgets.

We thank the reviewer for their comment. We already state that uPRM still exhibits hacking in some settings. In particular, for domain-specific Qwen2.5-Math policies, we do not observe hacking and all steps are completed, whereas for Qwen2.5-7B, we observe hacking with both uPRM and sPRM. Further, we kept all hyperparameters identical to PURE [1], and we allocate same budget to every run (4 epochs, 532 iterations). Hacking is identified from sharp transitions in reward, response length and KL, without using test accuracy. Finally, PURE authors trained their own PRM on Qwen2.5-Math-7B using PRM800K and concluded that reward hacking still occurred at step 270.

The term “unsupervised” may be inappropriate, as the supervision signal comes from an external model.

In our experiments, we did not rely on any external model. The joint score is computed by the same LLM that is LoRA-adapted into the PRM, so the method is self-training. This holds for all uPRM across families and parameter scales. The only inputs are a pretrained LLM and unlabeled trajectories. We use “unsupervised” in the label sense, i.e., no step annotations and no ground-truth final answers were used.

PRM800K appears outdated and insufficiently challenging.

We appreciate the reviewer’s concern. However, we want to point to the performance of 14B LLM-judge (Table D4) which shows that PRM800K is harder than the GSM8K and MATH splits of ProcessBench. Its F1 is 0.34 on PRM800K against 0.50 and 0.43 for GSM8K and MATH, and it locates the first error in only 25% of erroneous trajectories. Additionally, PRM800K remains the standard human-annotated corpus in recent PRM work [2,3]. Further, the ProcessBench [4] baseline Qwen2.5-Math-7B is fine-tuned on PRM800K’s training set and the PRMs in [1] is also trained on PRM800K.

  1. Cheng et al. Stop Summation: Min-Form Credit Assignment Is All Process Reward Model Needs for Reasoning, NeurIPS 2025
  2. Zhang et al. The Lessons of Developing Process Reward Models in Mathematical Reasoning. ACL 2025
  3. Khalifa et al. Process Reward Models That Think. TMLR 2026.
  4. Zheng et al. ProcessBench: Identifying Process Errors in Mathematical Reasoning, ACL 2025.

Thank you for the response. I remain concerned about weaknesses. I therefore maintain my original score.

Reviewer_jfSbOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes uPRM, a method for training process reward models without human supervision on ground-truth step annotations and final-answer verification. Instead, it uses LLM’s probabilities of correctness-marker tokens to score candidate first-error positions, then optimizes a PRM against this score using an entropy-regularized reinforcement learning objective. The main contribution is a framework for transferring an LLM’s correctness judgments into a step-level reward. For experimentation, the paper uses Qwen2.5-14B-Instruct to calculate the probabilities of correctness-marker tokens as training signal and also it as the base model of the PRM. The authors evaluated the PRM on ProcessBench, verifier-guided test-time scaling, and reinforcement learning. Results show improvements over LLMaaJ without CoT in selected settings, while remaining substantially below a strong supervised PRM on ProcessBench.

Critical strengths
  • Training without ground-truth step-level labels. The method provides a practical way to train a PRM using a frozen LLM’s probabilistic judgments, without requiring human step annotations. This reduces annotation requirements and could be useful in domains where reliable correctness labels are difficult to obtain.

  • In the reported RL experiments, uPRM is less susceptible to reward hacking than the matched supervised PRM and supports effective policy optimization. Besides, the paper claims that static error-localization accuracy (for example on ProcessBench) alone may not adequately predict a PRM’s usefulness as a training signal in RL.

Critical weaknesses
  • Limited evidence of competitiveness to stronger models. On ProcessBench, uPRM's performance is substantially below supervised Qwen2.5-Math-PRM-7B and does not outperform CoT-based LLM judging. Meanwhile, downstream experiments are restricted to Qwen2.5 and Llama-3.1/3.2 models, and the RL robustness comparison primarily uses the authors’ matched supervised PRM. Without evaluations on stronger, more recent large reasoning models and against stronger supervised PRMs under matched computational budgets, the evidence does not establish a compelling practical advantage.

  • The “fully unsupervised” framing overstates the independence of the training signal. Although the method requires no ground-truth correctness labels, it directly optimizes judgments supplied by a frozen pretrained LLM. Joint scoring may improve these judgments, but it provides no independent correctness signal to resolve the judge’s systematic errors. This makes the distinction from LLM-guided self-training less substantial than the framing suggests and limits confidence in its proposed applicability to domains where the underlying judge LLM is unreliable.

Questions for authors
  • Does uPRM’s advantage against reward hacking hold when compared with Qwen2.5-Math-PRM-7B, rather than only the matched sPRM?

  • CoT-PRM matches uPRM’s average ProcessBench F1. What are the relative training costs and downstream results of these two methods under matched settings? Does joint-score optimization offer a clear advantage over distilling the judge’s predictions?

1 comment
AuthorsOpenReview ↗

Response to Reviewer jfSb

Limited evidence of competitiveness to stronger models. On ProcessBench, uPRM's performance is substantially below supervised Qwen2.5-Math-PRM-7B ...

We thank the reviewer for their comment. We report supervised Qwen2.5-Math-PRM-7B as a reference skyline and not as a matched competitor. It is expected that it shows better results because it was trained on a large corpus and with human labels [1]. To isolate the contribution of the latter, we performed the matched comparison with sPRM, which shares our backbone, architecture and training corpus (PRM800K trajectories) but uses human step labels. Surprisingly, uPRM outperforms it by 5.6% on the OlympiadBench benchmark.

…and does not outperform CoT-based LLM judging.

CoT-PRM matches uPRM’s average ProcessBench F1. What are the relative training costs and downstream results of these two methods under matched settings? Does joint-score optimization offer a clear advantage over distilling the judge’s predictions?

It is important to note that the CoT-based LLM judging requires substantial computational costs compared to uPRM (1.7×1.7\times the FLOPs and >60×> 60\times the latency per judgment as shown in Section D.3). This highlights its much worse scalability in reinforcement learning or test-time scaling, where PRMs actually provide value. To provide inference-cost-matched baseline, we train a PRM that distills the CoT-based LLM judging labels and show that uPRM matches its performance on the first error localization (Table 1) and outperforms in reinforcement learning experiments (Table D8).

The “fully unsupervised” framing overstates the independence of the training signal. Although ... domains where the underlying judge LLM is unreliable.

Our main contribution is developing the first truly label-free PRM training method. We use “unsupervised” in the label sense: no step annotations and no ground-truth final answers are used, unlike prior step-label-free PRMs that still need outcome labels. We clarify that the aforementioned concern of inability to correct for systematic errors applies to the entire LLM-as-a-Judge paradigm that is widely adopted across the field and which fundamentally requires that the base LLM possesses sufficient capability to produce meaningful judgements. Nevertheless, we show that uPRM already improves upon the base model’s own judgements. Indeed, with an identical backbone, prompt and parametrization, uPRM yields 8.5% to 13.3% improvements across ProcessBench splits (Table 1).

Does uPRM’s advantage against reward hacking hold when compared with Qwen2.5-Math-PRM-7B, rather than only the matched sPRM?

We thank the reviewer for their question. As explained in the previous response, we aim to provide controlled ablations to isolate the contribution of uPRM. Thus, we compare against sPRM that differs from uPRM only in availability of labels. Thus, performing comparison with Qwen2.5-Math-PRM-7B also differs in backbone, data, and labelling pipeline, so any difference in hacking could not be attributed to a single factor. Furthermore, [3] trained their own PRM on Qwen2.5-Math-7B using PRM800K and concluded that reward hacking still occurred at step 270.

References:

  1. Zhang et al. The Lessons of Developing Process Reward Models in Mathematical Reasoning. ACL 2025
  2. Liu et al. Can 1B LLM Surpass 405B LLM? Rethinking Compute-Optimal Test-Time Scaling. arXiv preprint arXiv 2025
  3. Cheng et al. Stop Summation: Min-Form Credit Assignment Is All Process Reward Model Needs for Reasoning, NeurIPS 2025

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.