- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper studies the phenomenom that PRM to overcredit reasonable but incorrect reasoning steps. The authors attribute this to the data imbalance and pointwise cross-entropy training. The authors proposed PRISM, a training method using a stepwise contrastive ranking loss, temporal-lookahead hard negatives from future steps in the same trajectory, and a margin-based curriculum learning. The method is evaluated on PRMBench and ProcessBench and is subsequently used in Best-of-N selection, guided beam search, and GRPO policy optimization. The paper reports a substantial reduction in false-positive rate on PRMBench and improvements in downstream search settings.
Critical strengths
- The paper analyzes an important failure mode. False-positive process rewards can direct search or optimization toward convincing but incorrect trajectories.
- The empirical reduction in PRMBench false positives is meaningful.
Critical weaknesses
- The theoretical analysis (Theorem 2.1) only covers a simplified binary Best-of- setting where either the FPR or FNR is zero.
- The pairwise objective only ensures correct relative ranking and does not ensure that negative steps fall below the classification threshold. Also, the paper does not specify the operating threshold used to compute its headline FPR/FNR results.
- Temporal lookahead does not guarantee label validity: a future step may still be a correct continuation that skips multiple intermediate steps. The paper should add audit to estimate the fraction of invalid negatives.
- The authors should compare with more baseline methods such as class-balanced BCE, weighted BCE, and focal loss.
- The calculation of F1 scores seems to be incorrect for Omnimath in D.2 section.
Questions for authors
Please refer to the Weaknesses.
2 comments
Response to the Rebuttal (2/2)
W4) Additional comparisons with balanced BCE, weighted BCE, and focal loss
-
Our existing experiments already compares pairwise with that of data/class balancing BCE. Appendix C.3.1 (Table 6) compares pointwise BCE and pairwise training using identical positive-negative training examples, thereby controlling for class balance.
Pairwise training achieves an FPR of 61.75%, compared with 67.26% for pointwise BCE, demonstrating that its advantage extends beyond balancing the training distribution. Furthermore, Table 2 shows that pointwise training with data augmentation achieves only 67.21% FPR, compared with 47.13% for the complete PRISM framework.
-
Thus further evalutions on weighted BCE and focal loss would be redundant.
W5) Incorrect F1 scores for OmniMath
Thanks for pointing this out. By mistake we interchanged the columns: F1 and Correct Acc for OmniMath. We have corrected it in the manuscript and explained the formulas and conclusions properly.
Response to the Rebuttal (1/2)
We thank the reviewer for their constructive feedback and for recognizing both the importance of addressing false-positive bias in process reward models—particularly for downstream search and policy optimization—and the meaningful empirical reduction in false positives achieved by PRISM. We address each concern below with additional experimental evidence.
Although pairwise loss does not theoritically guarantee absolute score calibration, but it has been empirically validated to improve both PRM and policy alignment. Fig 7 shows improved separation of negative-step scores, while Figure 5b demonstrates a favorable FPR–FNR trade-off across thresholds. PRISM also reduces FPR from 69.34% to 47.13% (Table 2).
Benchmarks such as PRMBench and ProcessBench use binary classification thresholds to evaluate PRMs. However, in downstream policy alignment, PRMs primarily act as rankers rather than binary classifiers. Therefore, PRISM directly optimizes the relative reward margin between correct and incorrect steps (Eq. 5–6). Eventually PRISM shows that margin optimization leads to improved performance on PRM benchmarks which use threholding to evaluted.
Operating thereshold is the standard thresholds used in every benchmark which is 0.5.
Clarification: A lookahead step does not need to be factually incorrect in isolation; rather, we treat it as a lower-preference continuation relative to the human-annotated step at the current position.
While PRM800K provides high-quality step-level annotations as a reliable anchor, we acknowledge that some lookahead steps may represent valid shortcuts, compressed derivations, or alternative reasoning paths. However, given the sequential nature of reasoning tasks and PRM800K's high labeling quality, we hypothesize that a lookahead step is usually less appropriate at the current position than the human annotated positive step.
To test this, we randomly sampled 100 examples from each of the 4 Curriculum Learning (CL) bins in our augmented dataset and evaluated them using gpt-5.4-mini as a judge. Given the problem, prefix, annotated positive step, and lookahead step, the judge categorized whether the positive step was better, the lookahead step was better, or the comparison was ambiguous.
Table: Results: Human Step vs. Lookahead Step Evaluation
Takeaway: Across all CL bins, the annotated positive step is consistently preferred over the lookahead step in the vast majority of cases (73–81%), while the lookahead step was never judged as superior (0%). This confirms our assumption that lookahead steps function effectively as lower-preference targets.