- Rating
- 1
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 1
- Soundness
- 1
Read review
Paper summary
The paper introduces Unsupervised Process Reward Models (uPRMs). For each reasoning trajectory, the method samples a candidate first-error position and inserts correctness markers into the trajectory. A frozen LLM scores these assignments through the conditional probabilities of the markers, jointly processing multiple trajectories so that earlier examples provide context for subsequent judgments. The PRM is trained to maximize this score using entropy-regularized reinforcement learning, with an additional correction discouraging degenerate label assignments. Training uses reasoning trajectories from PRM800K without their step or outcome labels.
Critical strengths
The method removes both forms of task-specific correctness supervision during PRM training, drawing its signal from a frozen LLM’s marker probabilities. This is consequential because it offers a practical route to building process verifiers where reliable annotations or answer checkers are unavailable.
Critical weaknesses
-
The contribution of joint scoring is insufficiently isolated. A controlled comparison with independent scoring under otherwise identical training settings is missing.
-
Evidence for resistance to reward hacking remains limited. uPRM still exhibits hacking in some settings, and sensitivity to reward calibration and PURE hyperparameters remains unexplored. Reporting the last non-degenerate checkpoints also complicates comparisons under equal training budgets.
-
The term “unsupervised” may be inappropriate, as the supervision signal comes from an external model.
Questions for authors
PRM800K appears outdated and insufficiently challenging.
2 comments
Thank you for the response. I remain concerned about weaknesses. I therefore maintain my original score.
Response to Reviewer 69xu
We thank the reviewer for their comment. We would like to highlight that this comparison is identical to the LLM-as-a-Judge (Table 1), because the judge is the optimum of our training objective under independent scoring.
With independent scoring (N=1), Eq. (12) decouples across trajectories into
maxpEj∼p[S(j;s)]+γH(p)
whose maximizer is pγ∗(j∣τ)=∑kexp(S(k;s)/γ)exp(S(j;s)/γ)
By Eq. (6), qLLM(j∣τ)=exp(S(j;s)).
Thus, the maximizer above is just temperature-adjusted qLLM(j∣τ)1/γ, whose argmax is exactly the naive judge from Table 1 for any temperature γ. The gains brought by uPRM (Table 1) are therefore attributable to joint scoring.
We thank the reviewer for their comment. We already state that uPRM still exhibits hacking in some settings. In particular, for domain-specific Qwen2.5-Math policies, we do not observe hacking and all steps are completed, whereas for Qwen2.5-7B, we observe hacking with both uPRM and sPRM. Further, we kept all hyperparameters identical to PURE [1], and we allocate same budget to every run (4 epochs, 532 iterations). Hacking is identified from sharp transitions in reward, response length and KL, without using test accuracy. Finally, PURE authors trained their own PRM on Qwen2.5-Math-7B using PRM800K and concluded that reward hacking still occurred at step 270.
In our experiments, we did not rely on any external model. The joint score is computed by the same LLM that is LoRA-adapted into the PRM, so the method is self-training. This holds for all uPRM across families and parameter scales. The only inputs are a pretrained LLM and unlabeled trajectories. We use “unsupervised” in the label sense, i.e., no step annotations and no ground-truth final answers were used.
We appreciate the reviewer’s concern. However, we want to point to the performance of 14B LLM-judge (Table D4) which shows that PRM800K is harder than the GSM8K and MATH splits of ProcessBench. Its F1 is 0.34 on PRM800K against 0.50 and 0.43 for GSM8K and MATH, and it locates the first error in only 25% of erroneous trajectories. Additionally, PRM800K remains the standard human-annotated corpus in recent PRM work [2,3]. Further, the ProcessBench [4] baseline Qwen2.5-Math-7B is fine-tuned on PRM800K’s training set and the PRMs in [1] is also trained on PRM800K.