acceptodds
Under review as a conference paper at ICLR 2027

The Hidden Bias of Process Reward Models: PRISM for Rewarding the Right Reasoning

Abstract

Process Reward Models (PRMs) improve credit assignment for reasoning. However, we identify a hidden bias in PRMs caused by severe step-level training data imbalance. Standard cross-entropy training amplifies this bias, causing PRMs to overcredit plausible but incorrect steps and produce high false-positive rates. We show that these false positives have an asymmetric downstream effect: false negatives mainly slow exploration, whereas false positives actively steer Best-of- selection, guided decoding, and policy optimization toward flawed reasoning. This suggests that PRM training should shift from pointwise label fitting to reliable relative comparisons. Motivated by this insight, we introduce PRISM (Precision Ranking for Improved Step Modeling), a label-efficient training recipe that converts existing step annotations into matched positive–negative comparisons, optimizes a Step-Contrastive objective, augments hard negatives with a temporal lookahead strategy that requires no new human labels, and stabilizes learning with a difficulty-aware curriculum. Across PRMBench and ProcessBench, PRISM substantially reduces false positives and improves macro F1 over strong discriminative PRMs. Applied to policy optimization and search tasks, including guided decoding and Best-of- selection, it consistently improves accuracy and robustness. More broadly, trustworthy process supervision is not just about assigning high rewards, but about rewarding the right reasoning for the right reasons.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
Reward Hacking, Process Reward Models, Policy Misalignment

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_AuofOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies the phenomenom that PRM to overcredit reasonable but incorrect reasoning steps. The authors attribute this to the data imbalance and pointwise cross-entropy training. The authors proposed PRISM, a training method using a stepwise contrastive ranking loss, temporal-lookahead hard negatives from future steps in the same trajectory, and a margin-based curriculum learning. The method is evaluated on PRMBench and ProcessBench and is subsequently used in Best-of-N selection, guided beam search, and GRPO policy optimization. The paper reports a substantial reduction in false-positive rate on PRMBench and improvements in downstream search settings.

Critical strengths
  1. The paper analyzes an important failure mode. False-positive process rewards can direct search or optimization toward convincing but incorrect trajectories.
  2. The empirical reduction in PRMBench false positives is meaningful.
Critical weaknesses
  1. The theoretical analysis (Theorem 2.1) only covers a simplified binary Best-of-NN setting where either the FPR or FNR is zero.
  2. The pairwise objective only ensures correct relative ranking and does not ensure that negative steps fall below the classification threshold. Also, the paper does not specify the operating threshold used to compute its headline FPR/FNR results.
  3. Temporal lookahead does not guarantee label validity: a future step may still be a correct continuation that skips multiple intermediate steps. The paper should add audit to estimate the fraction of invalid negatives.
  4. The authors should compare with more baseline methods such as class-balanced BCE, weighted BCE, and focal loss.
  5. The calculation of F1 scores seems to be incorrect for Omnimath in D.2 section.
Questions for authors

Please refer to the Weaknesses.

2 comments
AuthorsOpenReview ↗

Response to the Rebuttal (1/2)

We thank the reviewer for their constructive feedback and for recognizing both the importance of addressing false-positive bias in process reward models—particularly for downstream search and policy optimization—and the meaningful empirical reduction in false positives achieved by PRISM. We address each concern below with additional experimental evidence.

W1) Theoretical analysis is limited to simplified settings

  1. We agree that Theorem 2.1 is a simplified abstraction and does not fully capture scalar-score Best-of-N selection. Its purpose is to isolate the fundamental asymmetry between false positives and false negatives, illustrating how incorrectly rewarded candidates can remain competitive even as the number of sampled solutions increases.
  2. However, Appendix C.1 (Eq. 15) already derives the general Best-of-N expression with both FPR and FNR nonzero. A nonzero FPR imposes a performance ceiling, whereas when FPR is zero, the model converges to perfect selection despite nonzero FNR. This establishes the central asymmetry beyond the two special cases.
  3. In practice, downstream policy optimization depends on relative reward rankings rather than binary classification, which directly motivates PRISM's step-contrastive objective (Eq. 5–6). Our empirical results further validate this motivation through consistent improvements in Best-of-N selection and guided decoding (Figure 6).

W2) Pairwise ranking does not guarantee correct classification thresholds

  1. Although pairwise loss does not theoritically guarantee absolute score calibration, but it has been empirically validated to improve both PRM and policy alignment. Fig 7 shows improved separation of negative-step scores, while Figure 5b demonstrates a favorable FPR–FNR trade-off across thresholds. PRISM also reduces FPR from 69.34% to 47.13% (Table 2).

  2. Benchmarks such as PRMBench and ProcessBench use binary classification thresholds to evaluate PRMs. However, in downstream policy alignment, PRMs primarily act as rankers rather than binary classifiers. Therefore, PRISM directly optimizes the relative reward margin between correct and incorrect steps (Eq. 5–6). Eventually PRISM shows that margin optimization leads to improved performance on PRM benchmarks which use threholding to evaluted.

  3. Operating thereshold is the standard thresholds used in every benchmark which is 0.5.

W3) Temporal lookahead may introduce invalid negative labels

Clarification: A lookahead step does not need to be factually incorrect in isolation; rather, we treat it as a lower-preference continuation relative to the human-annotated step at the current position.

While PRM800K provides high-quality step-level annotations as a reliable anchor, we acknowledge that some lookahead steps may represent valid shortcuts, compressed derivations, or alternative reasoning paths. However, given the sequential nature of reasoning tasks and PRM800K's high labeling quality, we hypothesize that a lookahead step is usually less appropriate at the current position than the human annotated positive step.

To test this, we randomly sampled 100 examples from each of the 4 Curriculum Learning (CL) bins in our augmented dataset and evaluated them using gpt-5.4-mini as a judge. Given the problem, prefix, annotated positive step, and lookahead step, the judge categorized whether the positive step was better, the lookahead step was better, or the comparison was ambiguous.

Table: Results: Human Step vs. Lookahead Step Evaluation

CL BinPositive Step Preferred over LookaheadAmbiguousLookahead Step Preferred over Positive
Bin 178%22%0%
Bin 277%23%0%
Bin 381%19%0%
Bin 473%27%0%

Takeaway: Across all CL bins, the annotated positive step is consistently preferred over the lookahead step in the vast majority of cases (73–81%), while the lookahead step was never judged as superior (0%). This confirms our assumption that lookahead steps function effectively as lower-preference targets.

AuthorsOpenReview ↗

Response to the Rebuttal (2/2)

W4) Additional comparisons with balanced BCE, weighted BCE, and focal loss

  1. Our existing experiments already compares pairwise with that of data/class balancing BCE. Appendix C.3.1 (Table 6) compares pointwise BCE and pairwise training using identical positive-negative training examples, thereby controlling for class balance.

    Pairwise training achieves an FPR of 61.75%, compared with 67.26% for pointwise BCE, demonstrating that its advantage extends beyond balancing the training distribution. Furthermore, Table 2 shows that pointwise training with data augmentation achieves only 67.21% FPR, compared with 47.13% for the complete PRISM framework.

  2. Thus further evalutions on weighted BCE and focal loss would be redundant.

W5) Incorrect F1 scores for OmniMath

Thanks for pointing this out. By mistake we interchanged the columns: F1 and Correct Acc for OmniMath. We have corrected it in the manuscript and explained the formulas and conclusions properly.

Reviewer_5qjtOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The paper asks whether process reward models (PRMs) over-score incorrect reasoning steps. PRISM combines a pairwise ranking loss for alternative steps sharing the same prefix, temporal-lookahead augmentation that uses future steps as negatives for earlier prefixes, and a curriculum based on positive–negative score margins. The paper then models false positives and false negatives in Best-of-N selection and evaluates the method on PRMBench, ProcessBench, inference-time search, and GRPO policy training.

On PRMBench, the reported FPR falls from 69.34% to 47.13% and PRMScore rises from 65.50 to 68.00, while FNR rises from 4.27% to 12.69%. The paper also reports gains on some downstream tasks. My main concern is whether these gains reflect better reasoning discrimination or simply a different scoring operating point or another experimental factor.

Critical strengths
  1. The error matters in each target use: overestimating incorrect steps can affect search pruning, answer selection, and policy training. Linking verifier error types to downstream decisions is a useful framing for process supervision.
  2. The recipe can be applied to existing PRMs and step annotations without collecting new human labels. The PRMBench and downstream results merit follow-up under controlled evaluation; if they hold, the low data-construction cost would make the method practical.
Critical weaknesses
  • W1: Unvalidated lookahead labels, contradictory curriculum filtering, and overclaimed theory. (1) Temporal-lookahead augmentation (Section 3.2) treats future steps as negatives for earlier prefixes without validation, risking penalizing valid shortcuts or compressed derivations. (2) While Section 3.1 and Appendix C.3 claim the pairwise loss flips reversed rankings (st+<st−s_t^+ < s_t^-), Section 3.3 retains only pairs with DHard=st+−st−≥0D_{\text{Hard}} = s_t^+ - s_t^- \ge 0 (Table 7). Discarding all DHard<0D_{\text{Hard}} < 0 pairs means PRISM never trains on actual ranking reversals, mainly widening existing positive margins. (3) Theorem 2.1's claim that false negatives only slow convergence requires α=0\alpha = 0; per Appendix C.1.1 (Eqs. 13, 16), when α>0\alpha > 0 the asymptotic ceiling is p(1−β)/[p(1−β)+(1−p)α]p(1-\beta)/[p(1-\beta)+(1-p)\alpha], so PRISM's nearly tripled FNR (β\beta: 4.27% →\to 12.69%) also lowers the ceiling.

  • W2: Contradictory Best-of-N and ProcessBench results. (1) In Figure 6, Best-of-N curves across PRMs diverge sharply at N=1N=1 (e.g., 33% to 58% on LiveCodeBench; 51% vs. 59% on MATH-500 OOD), whereas post-hoc selection over a shared candidate pool must coincide at N=1N=1—and indeed does coincide in Appendix Figure 9 (51.5% OOD, 73.3% ID), directly contradicting Figure 6. (2) Appendix D.2 claims "lowering FPR is accompanied by a decrease in TPR," yet the ProcessBench table shows the exact opposite from Baseline to CL3: TPR rises across all benchmarks (e.g., 67.0→72.667.0 \to 72.6 on MATH500) while TNR drops (91.1→82.091.1 \to 82.0, so FPR worsens from 8.9% to 18.0%), and OmniMath reports an F1 of 83.4 from 54.8 and 66.1 (harmonic mean: 59.92).(3) Table 5 specifies a "Rule-based math verifier reward" for GRPO, contradicting Section 4.2.2 and Table 3.

Questions for authors
  • Q1 (Best-of-N protocol and Figure 6 vs. Figure 9): Were the exact same candidate solutions reused across PRMs in Figure 6? Please explain why the N=1N=1 accuracies differ across verifiers in Figure 6 and why they disagree with the coinciding N=1N=1 points in Appendix Figure 9. Please also specify how step-level scores are aggregated into a trajectory score v(si)v(s_i) for weighted voting, and clarify how the guided beam-search x-axis ("Number of solutions") relates to beam width and step expansions.

  • Q2 (ProcessBench anomalies and GRPO reward): In Appendix D.2, why does TNR decrease (FPR increase) across curriculum stages in the table while the text claims FPR is lowered, and how is the OmniMath F1 of 83.4 computed from 54.8 and 66.1? In addition, please clarify whether GRPO uses the PRM, the rule-based math verifier listed in Table 5, or a combination. Would correcting these tables or configurations change the main conclusions?

  • Q3 (Lookahead labels and curriculum filtering): What fraction of temporal-lookahead negatives are genuinely invalid steps rather than valid compressed derivations? Furthermore, since Section 3.3 discards all pairs with DHard<0D_{\text{Hard}} < 0 (where the baseline PRM actually misranks the negative above the positive), how does the model learn to flip reversed rankings rather than merely rescaling already-correct margins?

  • Q4 (Controlled comparison): Under matched data (including the same 132k filtered curriculum pool), training order, and compute, does the pairwise loss outperform BCE or class-balanced BCE in FPR at a fixed TPR while retaining downstream gains on shared candidates? This would distinguish improved discrimination from an operating-point shift.

2 comments
AuthorsOpenReview ↗

Response to the Rebuttal (1/2)

We thank the reviewer for their detailed and constructive feedback. We appreciate the opportunity to clarify PRISM's temporal-lookahead strategy, curriculum design, theoretical assumptions, and evaluation methodology. We address each concern below with additional experimental evidence and manuscript corrections.

W1: Unvalidated lookahead labels, contradictory curriculum filtering, and overclaimed theory.

W1.1) Temporal-lookahead augmentation (Section 3.2) treats future steps as negatives for earlier prefixes without validation, risking penalizing valid shortcuts or compressed derivations.

  1. Clarification on label validity: We agree that a future step may be mathematically correct even when intermediate derivations are omitted. Our objective is not to classify such steps as factually incorrect, but to learn their relative preference compared with the human-annotated step at the current position. Given the sequential nature of reasoning tasks, we hypothesize that lookahead steps are generally less appropriate continuations than the annotated positive steps. We will clarify this distinction in Section 3.2 and Figure 1.

  2. Independent label audit: To validate this assumption, we evaluated 400 randomly sampled augmented pairs (100 per curriculum bin) using GPT-5.4-mini as an independent judge. The annotated positive step was preferred in 73–81% of cases, with 19–27% ambiguous comparisons and 0% preferring the lookahead step. These results support the use of lookahead steps as lower-preference training targets, while acknowledging that they are not necessarily logically incorrect.

Table: Human-Annotated Step vs. Lookahead Step

CL BinsPositive PreferredAmbiguousLookahead Preferred
Bin 178%22%0%
Bin 277%23%0%
Bin 381%19%0%
Bin 473%27%0%

W1.2) While Section 3.1 and Appendix C.3 claim the pairwise loss flips reversed rankings (st+−st−s^{+}_t - s^{-}_t), Section 3.3 retains only pairs with Dhard=st+−st−>=0D_{hard} = s^{+}_t - s^{-}_t >= 0 (Table 7). Discarding all Dhard<0D_{hard}<0 pairs means PRISM never trains on actual ranking reversals, mainly widening existing positive margins.

Negative-margin filtering: The curriculum excludes negative-margin pairs to avoid potentially unstable optimization during early preference learning. We acknowledge that this may exclude informative model errors. This filtering is an optimization-stability heuristic rather than a claim that such pairs are uninformative. The curriculum experiments in Appendix D.3 demonstrate the effectiveness of progressively introducing harder comparisons.

W1.3) Theorem 2.1's claim that false negatives only slow convergence requires α=0\alpha = 0; per Appendix C.1.1 (Eqs. 13, 16), when α>0\alpha > 0 the asymptotic ceiling is p(1−β)/[p(1−β)+(1−p)α]p(1-\beta)/[p(1-\beta)+(1-p)\alpha], so PRISM's nearly tripled FNR (β\beta: 4.27% 12.69%) also lowers the ceiling.

We agree that the Theorem 2.1's claim regarding FNR slows confergence applies specifically when FPR is zero. Its sole purpose is to isolate the fundamental asymmetry between false positives and false negatives.

Therefore, to show the practical and the general Best-of-N expression when both FPR and FNR nonzero we derive Eq 13, 16 in Appendix C.1 (Eq. 15).

Conclusion is: that a nonzero FPR imposes a performance ceiling, whereas when FPR is zero, the model converges to perfect selection despite nonzero FNR. This establishes the central asymmetry beyond the two special cases.

W2: Contradictory Best-of-N and ProcessBench results.

W2.1) In Figure 6, Best-of-N curves across PRMs diverge sharply at N=1N=1 (e.g., 33% to 58% on LiveCodeBench; 51% vs. 59% on MATH-500 OOD), whereas post-hoc selection over a shared candidate pool must coincide at N=1N=1—and indeed does coincide in Appendix Figure 9 (51.5% OOD, 73.3% ID), directly contradicting Figure 6.

The discrepancy arises from differences in the generator's sampled outputs across independent evaluation runs, rather than differences in PRM scoring at N=1.

For the statistical results in Appendix Figure 9, we used three fixed generation seeds shared across all PRMs, ensuring identical candidate generations and, consequently, identical performance at N=1.

For Figure 6 we used independent stochastic generation runs without enforcing a shared candidate pool. The reason is to keen BoN generations consistent with the generations in guided beam search (GBS). In GBS PRMs actively influence intermediate generation decisions, so different verifiers will produce different reasoning trajectories. Thus, the BoN results lead to different N=1 accuracies even though the PRM does not influence selection at N=1.

AuthorsOpenReview ↗

Response to the Rebuttal (2/2)

W2.2) Appendix D.2 claims "lowering FPR is accompanied by a decrease in TPR," yet the ProcessBench table shows the exact opposite from Baseline to CL3: TPR rises across all benchmarks (e.g., 67.0 -> 72.6 on MATH500) while TNR drops (91.1 -> 82.0, so FPR worsens from 8.9% to 18.0%), and OmniMath reports an F1 of 83.4 from 54.8 and 66.1 (harmonic mean: 59.92).

  1. OmniMath correction: The F1 and Correct Accuracy columns were inadvertently interchanged for OmniMath. This has been corrected in Appendix D.2, along with the metric definitions and interpretation.

  2. ProcessBench metric definitions: Importantly, ProcessBench does not report conventional step-level TPR/TNR/FPR/FNR. Instead, it evaluates the exact localization of the first erroneous step and the identification of entirely correct trajectories:

Error Accuracy = (# Erroneous trajectories with correctly identified first error) \ (# Total erroneous trajectories)

Correct Accuracy = (# Error-free trajectories correctly identified as error-free) \ (# Total error-free trajectories)

F1 = (2 * Error Accuracy * Correct Accuracy) \ (Error Accuracy + Correct Accuracy)

We have added these clarification in the appendix section along with the conclusion as mentioned below.

  1. Interpretation: PRISM improves Error Accuracy over the baseline across all four ProcessBench datasets, indicating stronger first-error localization. However, this comes with reduced Correct Accuracy potentially showing improved positive bias of the model, eventually resulting in mixed F1 performance.

Importantly, our central findings remain unchanged: PRISM substantially reduces step-level FPR on PRMBench (69.34% to 47.13%, Table 2) and consistently improves downstream Best-of-N selection and guided decoding (Figure 6).

W2.3) Table 5 specifies a "Rule-based math verifier reward" for GRPO, contradicting Section 4.2.2 and Table 3.

In Appendix B, GRPO (policy training) uses the reward from our trained PRM and not rule-based math-verifier reward. The step level PRM reward is converted into scalar trajectory reward by taking product of the step rewards. We have fixed the typo in the appendix B. Thanks a lot for pointing it out.

Q1 (Best-of-N protocol and Figure 6 vs. Figure 9): Were the exact same candidate solutions reused across PRMs in Figure 6? Please explain why the accuracies differ across verifiers in Figure 6 and why they disagree with the coinciding points in Appendix Figure 9. Please also specify how step-level scores are aggregated into a trajectory score for weighted voting, and clarify how the guided beam-search x-axis ("Number of solutions") relates to beam width and step expansions.

Please check answer to W2.

Q2 (ProcessBench anomalies and GRPO reward): In Appendix D.2, why does TNR decrease (FPR increase) across curriculum stages in the table while the text claims FPR is lowered, and how is the OmniMath F1 of 83.4 computed from 54.8 and 66.1? In addition, please clarify whether GRPO uses the PRM, the rule-based math verifier listed in Table 5, or a combination. Would correcting these tables or configurations change the main conclusions?

Please check answer to W2.

Q3 (Lookahead labels and curriculum filtering): What fraction of temporal-lookahead negatives are genuinely invalid steps rather than valid compressed derivations? Furthermore, since Section 3.3 discards all pairs with (where the baseline PRM actually misranks the negative above the positive), how does the model learn to flip reversed rankings rather than merely rescaling already-correct margins?

Please check answer to W1.

Q4 (Controlled comparison): Under matched data (including the same 132k filtered curriculum pool), training order, and compute, does the pairwise loss outperform BCE or class-balanced BCE in FPR at a fixed TPR while retaining downstream gains on shared candidates? This would distinguish improved discrimination from an operating-point shift.

  1. Existing matched-data comparison: Appendix C.3.1 (Table 6) already compares pairwise and pointwise BCE training using identical positive-negative examples, controlling for class balance. Pairwise training achieves 61.75% FPR versus 67.26% for pointwise BCE, demonstrating improvements beyond training-data rebalancing. Following results of Table 6, theorem C1 concludes that at any fixed recall/TPR, the BT model yields weakly smaller FPR.

  2. Augmented-data comparison: Table 2 further shows that pointwise training with augmentation achieves 67.21% FPR, whereas the complete PRISM framework achieves 47.13%.

Reviewer_H9GqOpenReview ↗
Rating
2(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2(+1)
Read review
Paper summary

The paper argues that positive-label imbalance causes process reward models to overcredit incorrect reasoning steps, with harmful downstream effects. PRISM fine-tunes PRMs using a pairwise ranking loss, additional negatives constructed from future steps of the same trajectory, and a margin-based curriculum. The paper evaluates step-level predictions, inference-time search, and downstream policy optimization.

Critical strengths
  1. The paper studies PRMs through their downstream use, rather than relying only on an aggregate verifier score. Evaluating both search and policy optimization is appropriate for the stated motivation.

  2. The training recipe reuses existing annotations, and the paper includes a pointwise-loss baseline trained on augmented data. The reported search results suggest that the resulting verifier may be useful, even though the current analysis does not adequately establish why.

Critical weaknesses
  1. Temporal lookahead does not provide reliable negative labels as described. A future step can be a valid continuation of an earlier prefix if it compresses intermediate algebra or states a logically implied conclusion. Section 3.2 explicitly treats writing a valid final root before intermediate simplifications as an invalid next step, but omission of those simplifications does not by itself make the step incorrect. Figure 1 similarly labels x=−5x=-5 as a hard negative after (x+2)2=9(x+2)^2=9, although it is a valid root; whether it is incomplete or inappropriate depends on what the step claims. Training on these pairs may teach a preference for a particular derivation granularity rather than better detection of incorrect reasoning. This concern is amplified by discarding negative-margin pairs according to the existing PRM, which can remove exactly the mistakes the method aims to correct.

  2. Several results do not support the stated general conclusions. Appendix D.2 describes a reduction in FPR accompanied by a decrease in TPR, but the table generally shows increasing TPR and decreasing TNR. For MATH500, TNR falls from 91.1 to 82.0, implying an FPR increase from 8.9% to 18.0%. Reported OmniMath F1 also falls from 83.4 to 72.6. These results conflict with the claimed consistent improvement on ProcessBench.

Questions for authors
  1. What conditions make a future step an invalid continuation? Can you provide an independent audit of the augmented labels and results excluding temporal-lookahead negatives?

  2. Please reconcile the ProcessBench table with the surrounding claims. How are TPR, TNR, and F1 defined and aggregated? In Table 2, why do some FNR and TPR entries not sum to 100%?

  3. Can you narrow the theoretical claims to the conditions actually analyzed, and test whether the conclusions hold for the continuous-score aggregation used in the search experiments?

  4. Appendix B lists a rule-based math verifier reward for GRPO, while Section 4.2.2 describes replacing the reward PRM. What reward was actually used, and how were process scores combined with any outcome reward?

2 comments
AuthorsOpenReview ↗

Response to the Rebuttal (1/2)

We thank the reviewer for their constructive feedback. We appreciate their recognition of PRISM's emphasis on downstream search and policy optimization, its label-efficient training approach, and its promising search results. These strengths reinforce the central motivation of our work. We address the concerns below with additional clarifications and experimental evidence.

W1) Temporal lookahead does not provide reliable negative labels as described. A future step can be a valid continuation of an earlier prefix if it compresses intermediate algebra or states a logically implied conclusion. Section 3.2 explicitly treats writing a valid final root before intermediate simplifications as an invalid next step, but omission of those simplifications does not by itself make the step incorrect. Figure 1 similarly labels (x=-5) as a hard negative after ((x+2)^2=9), although it is a valid root; whether it is incomplete or inappropriate depends on what the step claims. Training on these pairs may teach a preference for a particular derivation granularity rather than better detection of incorrect reasoning. This concern is amplified by discarding negative-margin pairs according to the existing PRM, which can remove exactly the mistakes the method aims to correct.

  1. Clarification on label validity: We agree that a future step may be mathematically correct even when intermediate derivations are omitted. Our objective is not to classify such steps as factually incorrect, but to learn their relative preference compared with the human-annotated step at the current position. Given the sequential nature of reasoning tasks, we hypothesize that lookahead steps are generally less appropriate continuations than the annotated positive steps. We will clarify this distinction in Section 3.2 and Figure 1.

  2. Independent label audit: To validate this assumption, we evaluated 400 randomly sampled augmented pairs (100 per curriculum bin) using GPT-5.4-mini as an independent judge. The annotated positive step was preferred in 73–81% of cases, with 19–27% ambiguous comparisons and 0% preferring the lookahead step. These results support the use of lookahead steps as lower-preference training targets, while acknowledging that they are not necessarily logically incorrect.

Table: Human-Annotated Step vs. Lookahead Step

CL BinsPositive PreferredAmbiguousLookahead Preferred
Bin 178%22%0%
Bin 277%23%0%
Bin 381%19%0%
Bin 473%27%0%
  1. Negative-margin filtering: The curriculum excludes negative-margin pairs to avoid potentially unstable optimization during early preference learning. We acknowledge that this may exclude informative model errors. This filtering is an optimization-stability heuristic rather than a claim that such pairs are uninformative. The curriculum experiments in Appendix D.3 demonstrate the effectiveness of progressively introducing harder comparisons.

W2) Several results do not support the stated general conclusions. Appendix D.2 describes a reduction in FPR accompanied by a decrease in TPR, but the table generally shows increasing TPR and decreasing TNR. For MATH500, TNR falls from 91.1 to 82.0, implying an FPR increase from 8.9% to 18.0%. Reported OmniMath F1 also falls from 83.4 to 72.6. These results conflict with the claimed consistent improvement on ProcessBench.

  1. OmniMath correction: We thank the reviewer for identifying the reporting error. The F1 and Correct Accuracy columns were inadvertently interchanged for OmniMath. This has been corrected in Appendix D.2, along with the metric definitions and interpretation.

  2. ProcessBench metric definitions: Importantly, ProcessBench does not report conventional step-level TPR/TNR/FPR/FNR. Instead, it evaluates the exact localization of the first erroneous step and the identification of entirely correct trajectories:

Error Accuracy = (# Erroneous trajectories with correctly identified first error) / (# Total erroneous trajectories)

Correct Accuracy = (# Error-free trajectories correctly identified as error-free) / (# Total error-free trajectories)

F1 = (2 * Error Accuracy * Correct Accuracy) / (Error Accuracy + Correct Accuracy )

  1. Interpretation: PRISM improves Error Accuracy over the baseline across all four ProcessBench datasets, indicating stronger first-error localization. However, this comes with reduced Correct Accuracy potentially showing improved positive bias in the model, eventually resulting in mixed F1 performance.

Importantly, our central findings remain unchanged: PRISM substantially reduces step-level FPR on PRMBench (69.34% to 47.13%, Table 2) and consistently improves downstream Best-of-N selection and guided decoding (Figure 6).

AuthorsOpenReview ↗

Response to the Rebuttal (2/2)

Q1) What conditions make a future step an invalid continuation? Can you provide an independent audit of the augmented labels and results excluding temporal-lookahead negatives?

Please check W1 answer above.

Please note that generating negatives through temporal-lookahead is "the" augmentation strategy. Furthermore, Table 2 already reports experiments without temporal-lookahead augmentation. Pairwise training using only existing annotated pairs reduces FPR from 69.34% to 61.13%, and adding curriculum learning further reduces it to 60.58%. Thus, the effectiveness of pairwise training is demonstrated independently of temporal-lookahead augmentation.

Q2) Please reconcile the ProcessBench table with the surrounding claims. How are TPR, TNR, and F1 defined and aggregated? In Table 2, why do some FNR and TPR entries not sum to 100%?

Please check W2 answer above.

Q3) Can you narrow the theoretical claims to the conditions actually analyzed, and test whether the conclusions hold for the continuous-score aggregation used in the search experiments?

  1. We agree that Theorem 2.1 is a simplified binary abstraction, not an exact characterization of scalar-score Best-of-N selection. Its purpose is to isolate the asymmetric effects of false-positive and false-negative predictions only.

  2. Testing ranking based search experiments: The downstream task is ranking (search experiments) and not classification. While all existing works evaluate PRMs as a classifier and compares it performance using classification metrics, we check its performance in ranking tasks by showing simulation in Figure 4. We basically show pairwise classification accuracy (ranking accuracy) of models trained using pointwise loss versus pairwise loss showing pairwise training is better for ranking tasks particularly in the lower margin region. Also, Figure 6 evaluates Best-of-N selection and guided decoding using PRM scores, demonstrating consistent improvements across mathematical reasoning and coding tasks.

These results provide empirical support for our ranking-based motivation, although they do not constitute a general theoretical guarantee for continuous-score aggregation which is out of scope of this work.

Q4) Appendix B lists a rule-based math verifier reward for GRPO, while Section 4.2.2 describes replacing the reward PRM. What reward was actually used, and how were process scores combined with any outcome reward?

In Appendix B, GRPO (policy training) uses the reward from our trained PRM and not rule-based math-verifier reward. The step level PRM reward is converted into scalar trajectory reward by taking product of the step rewards. We have fixed the typo in the appendix B. Thanks a lot for pointing it out.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.