acceptodds
Under review as a conference paper at ICLR 2027

Do Process Reward Models Fail to Detect Errors, or Fail to Decide? A Controlled Study of Operating Points

Abstract

Process reward models (PRMs) score intermediate reasoning steps, and judging the steps of a given solution turns those scores into decisions at a threshold. When a PRM misjudges steps, it is unclear whether its scores lack the information to detect errors or whether its decision boundary is simply misplaced. We separate these explanations with a controlled study that pairs a frozen-score audit of released PRMs on ProcessBench, PRMBench, and MedPRMBench with training interventions on 6,144 PRM800K chains that change either the clean-chain mixture or, on otherwise byte-identical inputs, only the class weights in the loss. We find that (1) released PRMs often discriminate far better than they decide: ReasonFlux-PRM and Qwen-Math reach nearly equal best shared-threshold ProcessBench Macro-F1 (73.36 vs. 73.50) yet differ by 28.35 points at the zero-margin boundary, which flags too few errors in 19 of 21 model–benchmark cells; (2) the training prior sets the logit origin rather than the ranking: loss-only interventions shift the operating point as the weighted-BCE optimum predicts (slope 0.987, ), shrink the best-to-fixed gap in all 12 paired Qwen3.5-4B runs, and reproduce 83–110% of the gap reduction from changing the mixture, while reverse weights reverse the effect; and (3) a source-set operating point transfers to a related benchmark but only partly to one with a higher error rate. Fixed-boundary evaluation thus conflates two failures that call for different fixes.

Primary area:
foundation or frontier models, including LLMs
Keywords:
process reward models, process supervision, first-error detection, decision thresholds, operating points, class weighting, data composition, loss reduction, score calibration, reasoning evaluation

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_8DdaOpenReview ↗
Rating
3(+1)
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

In this paper, the authors investigate whether poor step-level judgments by process reward models (PRMs) stem from insufficient discrimination between correct and incorrect reasoning steps or from misplaced decision thresholds / boundaries. The authors evaluate seven released PRMs across ProcessBench, PRMBench, and MedPRMBench, comparing fixed-threshold performance with the best performance achievable after selecting the threshold on evaluation labels. They find substantial differences between these measurements, with the fixed threshold usually flagging too few errors, suggesting that some apparently weak PRMs possess considerably better discrimination ability than their fixed-threshold scores indicate. To investigate the mechanism, the authors conduct controlled training experiments on Qwen3.5 PRMs, varying class weights while holding training examples and other conditions fixed. They find that the resulting logit shifts largely follow predictions from weighted binary cross-entropy theory, with relatively small changes in discrimination. Finally, they investigate threshold transfer from a labeled source-validation set to other benchmarks, finding near-optimal performance on PRMBench but larger residual errors on MedPRMBench.

Critical strengths
  • S1: The authors evaluate seven PRMs across three benchmarks and use threshold-independent metrics to support their findings. This gives confidence that the problem is not limited to a few models or datasets. The results could influence how future PRM evaluations are conducted, which strengthens the paper's significance.
  • S2: The authors isolate the effect of loss weighting by keeping the training data and setup fixed. The observed shifts closely match theoretical predictions, and reversing the weights reverses the effect. This makes the explanation more convincing than a purely observational analysis and is a strong reason to value the paper's scientific contribution.
Critical weaknesses
  • W1: The transfer experiments rely on a single validation set of 5,002 labeled chains, whose construction is not clearly described. Appendix G attributes the successful transfer to PRMBench partly to its similar error rate (15.8% versus 12.9% in validation), but does not test how transfer changes with different calibration samples, sizes, or error rates. Since all 12 transfer experiments use the same validation set, it is unclear how robust these results are to its particular composition.

  • W2: The finding that loss weighting shifts decision thresholds while largely preserving discrimination is not sufficiently established across models and training runs. The original Qwen checkpoints were not retained, and the rebuilt experiments show substantial variation in best achievable performance, even between identical runs. The only non-Qwen experiment uses one seed and shows an 8.73-point drop in best achievable performance. This limits confidence in how broadly the discrimination-preservation finding generalizes.

Questions for authors

Clarification Requests

  • QW1: How was the 5,002-chain source-validation set constructed? Have the authors tested whether the transfer results remain consistent when thresholds are selected using smaller or differently sampled source-validation sets? Table 15 investigates target-side calibration size, but does not appear to address the sensitivity of cross-benchmark transfer to source calibration.
  • QW2: Given that loss weighting substantially reduces discrimination in the Llama experiment, what evidence supports the conclusion that it primarily shifts decision thresholds across different model families? How much of the observed changes in discrimination can be distinguished from the run-to-run variation reported in Appendix G?

Future Work Suggestion

  • An interesting extension would be to explore whether synthetic step-level labels, such as those generated by independent verifiers (e.g., ScalePRM, arXiv:2512.03244), could support threshold calibration when ground-truth labels are unavailable.
4 comments
AuthorsOpenReview ↗

Calibration Robustness, Training Variability, and Synthetic Labels

8Dda

Thank you. We added source-sensitivity tests, paired cross-family experiments, and exploratory synthetic-label calibration.

W1/QW1. Source construction and sensitivity

The source set comes from PRM800K phase-2. We first filter exact, canonical, and TF-IDF near-matching questions against ProcessBench/PRMBench, then separate training and validation problem groups. The held-out set has 5,002 chains: 755 correct and 4,247 erroneous, with 12.9% erroneous steps in annotated prefixes. These chains do not train the controlled PRMs. With each PRM frozen, we select a threshold maximizing the harmonic mean of first-error localization on erroneous chains and acceptance of correct chains, using source labels only.

Across 17 checkpoints on three backbones, we use 300 draws per setting, sampling by problem group. The ranges below span backbone-average ProcessBench losses relative to the full source set:

Source chainsBackbone-average loss (points)
1,0000.02–0.19
5000.15–0.41
501.3–2.8

At 1,000 chains, varying the erroneous-chain share from 30% to 95% changes mean transfer by at most 0.58 points per checkpoint. On average, 500–1,000 chains retain performance close to the full pool; 50 incur larger losses. Within-class normalization in the selection criterion partly explains stability to chain proportions.

Exact/canonical source–training question matches are zero. Removing 100 chains from 22 near-match candidate questions leaves 44/46 thresholds unchanged; the largest threshold change is 0.035 and the largest transferred ProcessBench change is 0.59 points. Separately, one source threshold per released PRM, transferred unchanged to three benchmarks, improves 18/21 settings.

W2/QW2. Cross-family effects and run-to-run variation

We added three paired seeds each on Gemma-3-4B, GLM-4-9B, Llama-3.1-8B, and R1-Distill-Llama-8B. With Qwen3.5-9B, there are 31 interventions; Qwen's reverse arm has one seed. Pairs fix the data, initialization, order, and other settings; only class weights change. All 31 median paired step-logit shifts on ProcessBench follow the predicted direction (slope 0.94; R²=0.95).

To quantify the 4B variability raised in Appendix G, 18 baseline runs (three repeats per seed/loss setting) give a pooled within-setting Best SD of 1.77 points. We assess preservation using paired intervals. For forward weighting, jointly resampling problems and paired seeds gives:

BackboneMean ΔFixedMean ΔBestJoint 95% CI: ΔBest
Qwen9B+0.70+0.09[−0.68, +0.70]
Gemma+3.67+0.43[−0.87, +1.35]
GLM+1.66−0.17[−0.87, +0.37]
R1-Distill+3.95+0.33[−0.39, +1.45]
Llama-base−3.99−3.52[−5.16, −1.93]

The first four backbones improve mean Fixed, with mean Best changes below 0.5 points. Requiring both 95% intervals to lie within ±0.01 AUROC and ±1 Best point, 18/24 three-seed cells on these backbones meet the criterion with checkpoints fixed, and 13/24 with joint resampling. GLM meets it in all six cells under both analyses. Full cross-benchmark results include Gemma's greater medical-domain variability; mean stability and interval-based preservation remain distinct.

Llama-base loses Best in all three forward seeds; its joint ΔAUROC interval is [−0.0534, −0.0102]. The frozen-baseline boundary b=log(w−/w+) gives:

SeedBaseline @0.5Baseline @analytic boundaryClass-weighted @0.5
4226.7425.6222.66
7729.5429.4024.47
12328.9729.5026.15

The analytic boundary outperforms the class-weighted model in all three seeds. Score shifts generalize across families; paired intervals and frozen readouts identify the accompanying discrimination changes.

Synthetic step labels: an exploratory test of the suggestion

Following the ScalePRM suggestion [8], we use existing GenPRM-7B, R-PRM-7B, and Qwen3.5-9B judge outputs for all 25,697 steps of 3,400 ProcessBench chains. Each verifier averages correctness probabilities over eight samples per step, thresholds at 0.5, and the three verifiers vote. Only these synthetic labels select one threshold per frozen PRM; gold labels evaluate the selected rule.

ReadoutMean score across seven PRMs
Fixed 0.536.89
Labeled PRM800K source threshold54.32
Synthetic-label threshold55.40
Gold-label Best, diagnostic55.67

Synthetic calibration recovers about 99% of the summed Fixed–Best gaps, ending 0.27 points below mean Best. Six models improve over Fixed; already-aligned Qwen-Math changes by −0.28. This is exploratory, transductive calibration on the same target chains, without gold-label threshold tuning. Frozen scores are unchanged: the recovered performance comes from the boundary. We will cite ScalePRM and report all model results, label construction, and verification costs.

[8] ScalePRM, arXiv:2512.03244.

AuthorsOpenReview ↗

Synthetic-label calibration on PRMBench and MedPRMBench

Following your suggestion, we extended our ProcessBench experiment to subsets of PRMBench (900 chains) and MedPRMBench (500 chains). We labeled each step by majority vote among three verifiers and selected one threshold per frozen PRM using only these synthetic labels. We then evaluated against gold labels on the same chains. Relative to the fixed threshold of 0.5, mean step-level macro-F1 across seven PRMs rose from 54.21 to 59.97 on PRMBench (+5.76 points) and from 47.87 to 63.77 on MedPRMBench (+15.90 points). Six of seven PRMs improved on each subset. These gains closed 70.5% and 78.1% of the respective gaps to the gold-label oracle thresholds, compared with about 99% on ProcessBench. This shows that synthetic labels can improve threshold selection on the evaluated chains without using gold labels for calibration; it does not establish transfer to unseen chains.

Reviewer_8DdaOpenReview ↗

Response to Rebuttal

Thanks for the detailed reply. The source-sensitivity analysis and the extra backbones mostly settle my two concerns, and the synthetic-label calibration is a nice addition. The Llama-base exception is still real, so the preservation claim needs the qualification in the final version. Raising to 3 on that basis.

AuthorsOpenReview ↗

Thank you for the follow-up

Thank you for coming back to read the new experiments and for letting us know which parts addressed your concerns. We really appreciated hearing what you found convincing and where we still needed to be more precise. Your point about Llama-base was especially helpful, and we’ll make that exception explicit in the revision.

Reviewer_bjt8OpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper investigates whether process reward models fail because they cannot distinguish correct from incorrect reasoning steps or because their decision thresholds are poorly chosen. The authors evaluate seven models across three benchmarks and conduct controlled training experiments that vary class weights while holding the training examples fixed. The results suggest that, in the main Qwen experiments, class weighting primarily changes the operating point, with relatively little average improvement in performance at the best shared cutoff.

Critical strengths
  1. The paper offers a useful perspective on PRM evaluation. Separating discrimination from threshold selection helps explain why a model can perform poorly at the default cutoff despite assigning informative scores. The evaluation across several models and benchmarks makes this observation practically relevant.

  2. The controlled training experiments are the strongest component. Holding examples, initialization, and training order fixed helps isolate the effect of class weighting. Multiple seeds, alternative loss reductions, reverse weighting controls, and a larger model replication strengthen the analysis.

Critical weaknesses
  1. The contribution needs clearer distinction from prior work.

Threshold sensitivity in PRMs is already documented. R PRM (She et al., 2025, Appendix D, Figure 8) reports an 8.2 point difference between performance at the default cutoff and the optimal cutoff. Classical work on logit adjustment also explains how class weighting and class priors can shift decision boundaries (Menon et al., 2021). The manuscript acknowledges these connections, so the concern is the precise extent of the additional knowledge contributed.

The strongest distinctive contribution appears to be the controlled demonstration that changing class weights alone produces approximately predictable score shifts in PRMs. This contribution should be separated more clearly from the broader observation that default thresholds can be suboptimal. I would recommend adding more emphasized experiment to support this part of claim.

  1. The claim that ranking is preserved is broader than the evidence supports.

The small average change in performance at the best shared cutoff masks substantial variation. Across the main comparisons, changes range from a decrease of 2.95 points to an increase of 3.21 points (Appendix C, Table 12). In the Llama pilot, performance at the best cutoff decreases by 8.73 points, and AUROC falls from 0.7350 to 0.6563 (Appendix C.1).

These findings support a qualified conclusion: operating point shifts explain much of the observed behavior in the main Qwen setting, but class weighting can also change discrimination. The paper should define what magnitude of change counts as preservation and provide uncertainty estimates where possible. A lack of statistical significance alone does not establish equivalence.

  1. Potential overlap with PRMBench needs clarification.

PRMBench draws some correct solutions from both the training and test portions of PRM800K, while the controlled experiments also use PRM800K. The manuscript reports an overlap audit for the medical benchmark, but a corresponding audit for PRMBench is not clearly established.

This does not demonstrate contamination. It does leave uncertainty about the independence of the PRMBench evaluation and threshold transfer results. The authors should quantify any overlap and, if necessary, report results on the disjoint portion.

[1] She et al. (2025). R-PRM: Reasoning-Driven Process Reward Modeling. EMNLP. [2] Menon et al. (2021). Long-tail Learning via Logit Adjustment. ICLR.

Questions for authors
  1. What specific empirical conclusion goes beyond previously observed PRM threshold gaps and classical results on class weighting and logit adjustment?

  2. Is there overlap between the PRM800K examples used for training or threshold selection and PRMBench? If so, does removing it change the conclusions?

3 comments
AuthorsOpenReview ↗

Mechanism Evidence, Discrimination Changes, and Data Overlap

bjt8

Thank you. We clarify the contribution and data overlap.

Q1. What is established beyond threshold gaps and classical theory?

R-PRM documents threshold sensitivity, and logit adjustment supplies theoretical context [6,7]. Our contribution quantifies how that mechanism manifests in actual PRM training and decisions.

First, the predicted displacement is measured under controlled interventions across families. Five backbones from four families use the same 6,144 chains, with initialization, order, and training settings paired. All 31 median paired step-logit shifts on ProcessBench follow the predicted direction (slope 0.94; R²=0.95). Reverse weighting tests direction; Gemma's additional weight level tests dose response. Step-logit displacement and source-threshold displacement are reported as separate statistics.

Second, an analytic boundary approximates the class-weighted model's decisions. For r=w−/w+, we apply b=log r to the frozen baseline's correct-class logit margin and compare its predictions with those of the class-weighted model at b=0. This requires no parameter updates or target-label threshold search. Across the four backbones other than Llama-base, mean first-error decision agreement rises from 83% without adjustment to 91% with it (adjusted range 87%–97%).

In 24 answer-replacement settings across Qwen9B, Gemma and R1-Distill (12 on earlier pools; 12 on fresh MATH/GSM8K problems), the frozen baseline with the analytic boundary and the class-weighted model differ by at most 0.23 percentage points. With checkpoints fixed, all problem-bootstrap 90% intervals lie within ±1 percentage point. On fresh GSM8K with the Llama generator, class weighting improves Gemma's accuracy by 0.42 percentage points; the analytic boundary reproduces this gain to within about 0.02 percentage points.

Third, Fixed gains exceed changes in diagnostic Best on four backbones. Three-seed means for forward weighting on ProcessBench are:

BackboneΔFixedΔBest
Qwen3.5-9B+0.70+0.09
Gemma-3-4B+3.67+0.43
GLM-4-9B+1.66−0.17
R1-Distill-Llama-8B+3.95+0.33
Llama-3.1-8B base−3.99−3.52

Together with the frozen-readout control, these four results support operating-point adjustment as an important explanation of decision gains. Llama-base is a clear exception: all three seeds reduce Best; its joint 95% interval is [−5.16, −1.93]. Reading its frozen baseline at the analytic boundary outperforms the class-weighted model by 2.95–4.93 points across the three seeds, illustrating the practical value of separating the two effects.

To assess preservation, both 95% intervals must lie within ±0.01 step AUROC and ±1 Best point. Best is the diagnostic test-label optimum over one shared threshold. For the same three-seed cells:

BackbonesCheckpoints fixed; resample problemsJoint problem/seed resampling
Qwen9B, Gemma, GLM, R1-Distill18/2413/24
Llama-base1/60/6

GLM meets the criterion in all six cells under both analyses. Including three single-seed Qwen reverse cells yields 20/27 in the first row's conditional analysis only. Each cell is a backbone/weight-arm/benchmark setting; joint intervals use three seeds. Full results include Gemma's greater MedPRMBench variability. Small mean changes and interval-based preservation remain distinct.

Q2. Is there PRM800K–PRMBench overlap, and does exclusion change the findings?

At construction, PRM800K phase-2 data were filtered against ProcessBench/PRMBench questions using exact, canonical, and TF-IDF near-match checks, then split by problem group. The 5,002-chain source set comes from this filtered pool. Benchmark question text is used for exclusion, not benchmark labels for training or calibration.

A new audit covers the fixed-support set, both mixtures, the 20K training set, and the source set. Exact/canonical question matches are zero in every file. The new screen flags near-match candidates, so we apply a conservative evaluation-side exclusion: remove PRMBench records with question cosine≥0.7 or a verbatim shared step of at least 40 characters. This removes 1,168/6,211 records. Across the 12 original 4B interventions:

PRMBench cohortFixedSourceBestBest−Fixed
Full: 6,21166.3866.5966.720.34
Filtered: 5,04366.1366.2966.440.31

Gaps use unrounded scores. Best−Fixed remains similar (0.34→0.31), as does Best−Source (0.13→0.14). Separately, removing 100 source chains from 22 source/train near-match candidate questions leaves thresholds unchanged in 44/46 runs; the largest transferred ProcessBench change is 0.59 points. Construction filters, audit candidates, and exclusion results will be provided together.

References: [6] R-PRM, EMNLP 2025, §5.4 and Appendix D, Fig. 8. [7] Menon et al., Long-tail Learning via Logit Adjustment, ICLR 2021.

AuthorsOpenReview ↗

PRMBench overlap and discrimination on ProcessBench

For each of the 46 checkpoints from our five controlled backbones, we used the same source-selected threshold to evaluate PRMBench before and after excluding possible matches to our training or calibration data. Relative to the fixed threshold of 0.5, the mean gain was 1.30 step macro-F1 points on the 6,211 evaluated records and 1.06 points on the 5,043 retained after filtering.

Under forward weighting on ProcessBench, four of the five backbones scored higher at the default threshold of 0.5, with three-seed mean F1 gains ranging from 0.70 to 3.95 points. For each checkpoint, Best is the highest ProcessBench first-error macro-F1 obtained by selecting a single threshold on the evaluation labels and applying it across all four subsets. We estimated 95% confidence intervals by jointly resampling problems and paired training seeds. For each of the four backbones, the lower bound for ΔBest was above −1 point. Two intervals extended above +1: mean ΔBest was +0.43 for Gemma-3-4B (95% CI [−0.87, +1.35]) and +0.33 for R1-Distill-Llama-8B (95% CI [−0.39, +1.45]). For all four, the interval for the change in prefix step AUROC lay within ±0.01. For Llama-3.1-8B base, prefix step AUROC fell by 0.0276 on average (95% CI for the change: −0.0534 to −0.0102).

AuthorsOpenReview ↗

Follow-up on PRMBench overlap and discrimination

Thank you again for raising these points. In the additional analysis, the same source-selected thresholds still improved PRMBench step macro-F1 by 1.06 points over 0.5 after we excluded potential overlaps. The ProcessBench analysis also shows where the boundary-shift explanation falls short: Llama-base lost discrimination under forward weighting. If either point would benefit from clarification before the discussion closes, we’d be happy to follow up.

Reviewer_qEVsOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper separates two reasons a process reward model (PRM) may make poor step judgments: its scores may rank correct and incorrect steps poorly, or a shared decision threshold may be poorly placed. The authors audit seven released PRMs on ProcessBench, PRMBench, and MedPRMBench, comparing a fixed zero-margin reader with the best shared threshold on the test labels (clearly marked as diagnostic) and with step AUROC. They then train Qwen3.5 PRMs on the same 6,144 PRM800K chains while changing only the loss class weights, with data-mixture controls. They report that many fixed-boundary scores improve substantially at another threshold, and that weighting shifts the operating point in the predicted direction while mostly preserving ranking. The effect is strongest for Qwen; the 9B replication is partial, the single-seed Llama pilot also changes discrimination, and source thresholds transfer only partly to a benchmark with a different error rate. A fixed-pool MATH-500 check finds no significant accuracy gain.

Critical strengths
  • The paper makes a useful distinction between discrimination and decisions. Frozen-score threshold sweeps, AUROC, and explicit labeling of the test-set optimum as diagnostic help separate ranking quality from performance at a chosen boundary.

  • The Qwen intervention is well controlled. The forward comparisons hold the 6,144 training chains and other training choices fixed while varying class weights; the observed threshold shifts track the weighted-BCE prediction, and a reverse-weight control moves in the opposite direction. This is credible evidence for the mechanism within that experimental setting.

Critical weaknesses
  • The released-model audit’s fixed reference is not established as the intended decision rule for those models. The paper notes that an author-endorsed zero-margin or 0.5 rule is not established for any of the seven audited PRMs. Consequently, a large gap between that reference and the test-label optimum shows sensitivity to the threshold, but does not by itself establish that a released PRM fails at its intended use. This limits the force of the headline conclusion about released PRMs “failing to decide.”

  • The generality and practical significance of the controlled finding remain uncertain. The main multi-seed evidence is from the Qwen family; the second-backbone result is a single-seed pilot in which the forward intervention also reduces AUROC and Best. Since the weighting effect follows established weighted-BCE theory, broader evidence that it yields a useful PRM decision improvement would strengthen the contribution. The fixed-pool MATH-500 check does not establish a downstream accuracy gain.

Questions for authors
  • Do any of the seven released PRMs have a documented, author-recommended 0.5 or zero-margin decision rule? If not, could the paper frame the public-model results more narrowly as threshold sensitivity relative to a standardized convention?

  • Can the authors provide multi-seed evidence for the fixed-support loss-only intervention on a second backbone, or clarify that the generalization claim is limited to the Qwen family given the Llama pilot’s ranking changes?

2 comments
AuthorsOpenReview ↗

Evaluation Protocols, Cross-Family Evidence, and Downstream Results

Thank you for these questions.

1. Is 0.5 an author-recommended threshold?

We use 0.5 as a common reference, not as a general recommendation from the model authors. There are task-specific precedents. The ProcessBench authors use binary step predictions for their Qwen2.5-Math-7B-PRM800K baseline. Choosing the more likely of two classes gives a 0.5 boundary. The later Qwen PRM paper uses the same first-error protocol. PRMBench's Qwen, Math-Shepherd and RLHFlow adapters default to 0.5. ProcessBench, Qwen PRM, PRMBench adapters.

Universal-PRM selects its threshold on validation data, and ReasonFlux uses continuous rewards for data selection and Best-of-N. We will describe the audit in the abstract as measuring sensitivity to a common threshold and give each model's published protocol in the paper. Universal-PRM, ReasonFlux-PRM.

Our Qwen3.5-9B, trained on 20K PRM800K chains, uses 0.5 throughout (Fixed). Best is the highest score obtained with one threshold shared across all problems. We choose it separately for each checkpoint and benchmark using target labels, so it is diagnostic.

BenchmarkOurs: FixedHighest Fixed among seven audited PRMsOurs: Best−Fixed
ProcessBench75.1973.100.31
PRMBench68.4865.470.07
MedPRMBench70.8862.645.76

We selected one threshold per frozen PRM on 5,002 labeled PRM800K phase-2 chains. Applied unchanged, these thresholds improved on 0.5 in 18 of 21 model–benchmark pairs. We screened source questions against ProcessBench and PRMBench and kept them separate from our controlled training questions.

2. Does the effect of class weighting extend beyond Qwen?

We repeated the experiment on Gemma, GLM, Llama-base and R1-Distill with three paired seeds each. Each pair used the same 6,144 chains, initialization, data order and training settings. Only the class weights varied. With Qwen, this gives 31 interventions across five backbones from four families. Qwen's reverse arm used one seed.

On ProcessBench, all 31 median paired step-logit shifts followed the predicted direction (slope 0.94, R² = 0.95). The table gives mean changes in F1 over three seeds under forward weighting.

BackboneΔFixedΔBest
Qwen3.5-9B+0.70+0.09
Gemma-3-4B+3.67+0.43
GLM-4-9B+1.66−0.17
R1-Distill-Llama-8B+3.95+0.33
Llama-3.1-8B base−3.99−3.52

For the first four backbones, mean F1 at 0.5 rose by 0.70–3.95 points, while mean Best changed by less than 0.5 points in absolute value. For each, the 95% interval for the change in prefix step AUROC lay within ±0.01 when we resampled both problems and paired seeds. Llama-base also showed the predicted score shift, but Best fell in all three seeds. Its joint 95% intervals were [−5.16, −1.93] for ΔBest and [−0.0534, −0.0102] for the change in prefix step AUROC. The revised appendix will give the cross-benchmark intervals.

3. Can the operating point affect final-answer accuracy?

The fixed-pool MATH-500 check found no significant accuracy gain. In offline answer replacement, the threshold determines whether to keep the initial answer or use a preassigned alternative.

Threshold selection with frozen PRMs. We chose a threshold for each PRM–generator pair on 500 MATH development problems, then tested it on 1,500 previously unused problems (718 MATH-test, 782 MATH-train). For ReasonFlux with Llama, accuracy rose from 58.36% to 59.47% (+1.11 percentage points, paired problem-bootstrap 95% CI [+0.62, +1.62]). The table shows all four prespecified comparisons with 0.5.

PRM / generatorΔaccuracy (pp)Significant after Holm correction
Qwen-Math / Qwen-Math−0.38No
Qwen-Math / Llama+0.62No
ReasonFlux / Qwen-Math0.00No
ReasonFlux / Llama+1.11Yes

Qwen-Math denotes Qwen2.5-Math-PRM-7B as verifier and Qwen2.5-Math-7B-Instruct as generator. Llama denotes Llama-3.1-8B-Instruct.

Class weighting. On 1,500 held-out GSM8K problems (from the training split) with the Llama generator, weighting raised Gemma's mean accuracy at 0.5 by about 0.42 percentage points. We set the frozen baseline's correct-class logit boundary to log(w−/w+), using only the loss weights for incorrect and correct steps. Its accuracy came within about 0.02 percentage points of the class-weighted model's. Across 24 answer-replacement settings for Qwen9B, Gemma and R1-Distill (12 on earlier pools and 12 on fresh MATH/GSM8K problems), the difference was at most 0.23 percentage points. With checkpoints fixed, all 24 paired 90% problem-bootstrap intervals lay within ±1 percentage point.

AuthorsOpenReview ↗

Follow-up on the cross-backbone results

Thank you again for your questions. As the discussion wraps up, we wanted to highlight one result from our response: on ProcessBench, forward weighting improved F1 at the common 0.5 threshold by 0.70–3.95 points across four backbones, while mean F1 at the best shared threshold changed by less than 0.5 points in either direction for each. If you have any remaining questions about the evaluation protocol or the cross-backbone results, we’d be happy to clarify.

Comments on the paper

2 comments
AuthorsOpenReview ↗

General Response

We thank the reviewers and Area Chair for their feedback. Our additional experiments support operating-point adjustment as an important explanation of PRM decision gains. We summarize the main results below.

1. Controlled evidence across families. We evaluate 31 paired class-weighting interventions across five backbones from four families, using the same 6,144 training chains and matching initialization, data order, and other training settings. All arms use three paired seeds except Qwen9B's single-seed reverse arm. On ProcessBench, the median paired step-logit shift follows the predicted direction in all 31 interventions (slope 0.94; R² = 0.95). Under forward weighting, Qwen9B, Gemma, GLM, and R1-Distill improve mean Fixed by 0.70–3.95 points, with mean Best changes below 0.5 points in magnitude. Best is the diagnostic maximum over one shared threshold selected using target labels. Llama-base shows an accompanying discrimination loss: mean ΔBest is −3.52, with joint problem/seed 95% CI [−5.16, −1.93].

For the first four backbones, we assess preservation across 24 combinations of backbone, weighting arm, and benchmark, each evaluated with three paired seeds. Of these, 18/24 meet the criterion with checkpoints fixed, and 13/24 with joint problem/seed resampling. The criterion requires both 95% intervals to lie within ±0.01 step AUROC and ±1 Best point.

2. An analytic boundary reproduces downstream behavior. We apply b = log(w−/w+) to the frozen baseline's correct-class logit margin and compare the resulting decisions with those of the class-weighted model at b = 0. Across 24 answer-replacement settings on Qwen9B, Gemma, and R1-Distill, accuracy differences are at most 0.23 percentage points. These comprise 12 settings on earlier pools and 12 on fresh MATH/GSM8K problems. With checkpoints fixed, all paired problem-bootstrap 90% intervals for the accuracy difference lie within ±1 percentage point. On fresh GSM8K with the Llama generator, class weighting improves Gemma's accuracy by 0.42 percentage points, and the analytic boundary reproduces this gain to within about 0.02 percentage points. The analytic readout uses no parameter updates or target-label threshold search.

3. Calibration yields practical gains with frozen PRMs. Selecting one threshold per PRM on 5,002 labeled PRM800K source chains and transferring it unchanged to all three benchmarks improves over 0.5 in 18/21 released-model settings. In downstream answer replacement, selecting a threshold on 500 development problems disjoint from evaluation improves ReasonFlux with the Llama-3.1-8B-Instruct generator from 58.36% to 59.47% accuracy on 1,500 MATH problems not previously used in this project. The gain is +1.11 percentage points (paired problem-bootstrap 95% CI [+0.62, +1.62]) and remains significant after Holm correction over four prespecified primary comparisons. All four primary comparisons and the MATH cohort construction are reported in our response to Reviewer qEVs.

4. Synthetic labels provide an additional calibration route. Using aggregated step judgments from three verifiers, synthetic-label calibration raises the mean ProcessBench score across seven frozen PRMs from 36.89 to 55.40, compared with diagnostic Best of 55.67. It recovers about 99% of the summed Best−Fixed gaps. Six models improve over Fixed; Qwen-Math, which already has a small Best−Fixed gap, changes by −0.28 points. On subsets of PRMBench (900 chains) and MedPRMBench (500 chains), mean step-level Macro-F1 improved by 5.76 and 15.90 points over the fixed 0.5 threshold, respectively, with six of seven PRMs improving on each subset. In all three experiments, we used synthetic labels to select thresholds and gold labels only to evaluate them on the same chains; these results do not establish transfer to unseen chains.

Our reviewer-specific replies also address repeated-run variability, sensitivity to the size and composition of the source set, and the effects of excluding potential overlaps. Together, these results clarify when decision gains can be reproduced by adjusting the boundary and when class weighting also changes discrimination.

AuthorsOpenReview ↗

Summary of the Author–Reviewer Discussion

Thank you to the reviewers and the Area Chair for the discussion. The reviews highlighted a distinction at the heart of our study: how well a PRM distinguishes correct from incorrect steps can differ from how well it makes decisions at a fixed threshold. Three results from our General Response and follow-up analyses clarify that distinction:

  1. Threshold calibration after overlap filtering. Using 0.5 as a common reference, source-selected thresholds improved mean step-level macro-F1 on PRMBench by 1.06 points after we excluded potential matches to training or calibration data.
  2. Class weighting across backbones. On ProcessBench, forward weighting improved F1 at 0.5 by 0.70–3.95 points across four of five backbones. For each of those four, the 95% confidence interval for the change in step AUROC was within ±0.01. Llama-base was the exception, with a mean AUROC drop of 0.0276.
  3. Calibration with synthetic labels. On subsets of PRMBench and MedPRMBench, thresholds selected with synthetic labels improved six of seven frozen PRMs on each subset. We used gold labels only to evaluate performance on those same chains.

We will revise the abstract to describe 0.5 as a common reference and to distinguish the predictable shift in the operating point from changes in discrimination, including the decline for Llama-base.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.