- Rating
- 3(+1)
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
In this paper, the authors investigate whether poor step-level judgments by process reward models (PRMs) stem from insufficient discrimination between correct and incorrect reasoning steps or from misplaced decision thresholds / boundaries. The authors evaluate seven released PRMs across ProcessBench, PRMBench, and MedPRMBench, comparing fixed-threshold performance with the best performance achievable after selecting the threshold on evaluation labels. They find substantial differences between these measurements, with the fixed threshold usually flagging too few errors, suggesting that some apparently weak PRMs possess considerably better discrimination ability than their fixed-threshold scores indicate. To investigate the mechanism, the authors conduct controlled training experiments on Qwen3.5 PRMs, varying class weights while holding training examples and other conditions fixed. They find that the resulting logit shifts largely follow predictions from weighted binary cross-entropy theory, with relatively small changes in discrimination. Finally, they investigate threshold transfer from a labeled source-validation set to other benchmarks, finding near-optimal performance on PRMBench but larger residual errors on MedPRMBench.
Critical strengths
- S1: The authors evaluate seven PRMs across three benchmarks and use threshold-independent metrics to support their findings. This gives confidence that the problem is not limited to a few models or datasets. The results could influence how future PRM evaluations are conducted, which strengthens the paper's significance.
- S2: The authors isolate the effect of loss weighting by keeping the training data and setup fixed. The observed shifts closely match theoretical predictions, and reversing the weights reverses the effect. This makes the explanation more convincing than a purely observational analysis and is a strong reason to value the paper's scientific contribution.
Critical weaknesses
-
W1: The transfer experiments rely on a single validation set of 5,002 labeled chains, whose construction is not clearly described. Appendix G attributes the successful transfer to PRMBench partly to its similar error rate (15.8% versus 12.9% in validation), but does not test how transfer changes with different calibration samples, sizes, or error rates. Since all 12 transfer experiments use the same validation set, it is unclear how robust these results are to its particular composition.
-
W2: The finding that loss weighting shifts decision thresholds while largely preserving discrimination is not sufficiently established across models and training runs. The original Qwen checkpoints were not retained, and the rebuilt experiments show substantial variation in best achievable performance, even between identical runs. The only non-Qwen experiment uses one seed and shows an 8.73-point drop in best achievable performance. This limits confidence in how broadly the discrimination-preservation finding generalizes.
Questions for authors
Clarification Requests
- QW1: How was the 5,002-chain source-validation set constructed? Have the authors tested whether the transfer results remain consistent when thresholds are selected using smaller or differently sampled source-validation sets? Table 15 investigates target-side calibration size, but does not appear to address the sensitivity of cross-benchmark transfer to source calibration.
- QW2: Given that loss weighting substantially reduces discrimination in the Llama experiment, what evidence supports the conclusion that it primarily shifts decision thresholds across different model families? How much of the observed changes in discrimination can be distinguished from the run-to-run variation reported in Appendix G?
Future Work Suggestion
- An interesting extension would be to explore whether synthetic step-level labels, such as those generated by independent verifiers (e.g., ScalePRM, arXiv:2512.03244), could support threshold calibration when ground-truth labels are unavailable.
4 comments
Synthetic-label calibration on PRMBench and MedPRMBench
Following your suggestion, we extended our ProcessBench experiment to subsets of PRMBench (900 chains) and MedPRMBench (500 chains). We labeled each step by majority vote among three verifiers and selected one threshold per frozen PRM using only these synthetic labels. We then evaluated against gold labels on the same chains. Relative to the fixed threshold of 0.5, mean step-level macro-F1 across seven PRMs rose from 54.21 to 59.97 on PRMBench (+5.76 points) and from 47.87 to 63.77 on MedPRMBench (+15.90 points). Six of seven PRMs improved on each subset. These gains closed 70.5% and 78.1% of the respective gaps to the gold-label oracle thresholds, compared with about 99% on ProcessBench. This shows that synthetic labels can improve threshold selection on the evaluated chains without using gold labels for calibration; it does not establish transfer to unseen chains.
Response to Rebuttal
Thanks for the detailed reply. The source-sensitivity analysis and the extra backbones mostly settle my two concerns, and the synthetic-label calibration is a nice addition. The Llama-base exception is still real, so the preservation claim needs the qualification in the final version. Raising to 3 on that basis.
Thank you for the follow-up
Thank you for coming back to read the new experiments and for letting us know which parts addressed your concerns. We really appreciated hearing what you found convincing and where we still needed to be more precise. Your point about Llama-base was especially helpful, and we’ll make that exception explicit in the revision.
Calibration Robustness, Training Variability, and Synthetic Labels
8Dda
Thank you. We added source-sensitivity tests, paired cross-family experiments, and exploratory synthetic-label calibration.
W1/QW1. Source construction and sensitivity
The source set comes from PRM800K phase-2. We first filter exact, canonical, and TF-IDF near-matching questions against ProcessBench/PRMBench, then separate training and validation problem groups. The held-out set has 5,002 chains: 755 correct and 4,247 erroneous, with 12.9% erroneous steps in annotated prefixes. These chains do not train the controlled PRMs. With each PRM frozen, we select a threshold maximizing the harmonic mean of first-error localization on erroneous chains and acceptance of correct chains, using source labels only.
Across 17 checkpoints on three backbones, we use 300 draws per setting, sampling by problem group. The ranges below span backbone-average ProcessBench losses relative to the full source set:
At 1,000 chains, varying the erroneous-chain share from 30% to 95% changes mean transfer by at most 0.58 points per checkpoint. On average, 500–1,000 chains retain performance close to the full pool; 50 incur larger losses. Within-class normalization in the selection criterion partly explains stability to chain proportions.
Exact/canonical source–training question matches are zero. Removing 100 chains from 22 near-match candidate questions leaves 44/46 thresholds unchanged; the largest threshold change is 0.035 and the largest transferred ProcessBench change is 0.59 points. Separately, one source threshold per released PRM, transferred unchanged to three benchmarks, improves 18/21 settings.
W2/QW2. Cross-family effects and run-to-run variation
We added three paired seeds each on Gemma-3-4B, GLM-4-9B, Llama-3.1-8B, and R1-Distill-Llama-8B. With Qwen3.5-9B, there are 31 interventions; Qwen's reverse arm has one seed. Pairs fix the data, initialization, order, and other settings; only class weights change. All 31 median paired step-logit shifts on ProcessBench follow the predicted direction (slope 0.94; R²=0.95).
To quantify the 4B variability raised in Appendix G, 18 baseline runs (three repeats per seed/loss setting) give a pooled within-setting Best SD of 1.77 points. We assess preservation using paired intervals. For forward weighting, jointly resampling problems and paired seeds gives:
The first four backbones improve mean Fixed, with mean Best changes below 0.5 points. Requiring both 95% intervals to lie within ±0.01 AUROC and ±1 Best point, 18/24 three-seed cells on these backbones meet the criterion with checkpoints fixed, and 13/24 with joint resampling. GLM meets it in all six cells under both analyses. Full cross-benchmark results include Gemma's greater medical-domain variability; mean stability and interval-based preservation remain distinct.
Llama-base loses Best in all three forward seeds; its joint ΔAUROC interval is [−0.0534, −0.0102]. The frozen-baseline boundary b=log(w−/w+) gives:
The analytic boundary outperforms the class-weighted model in all three seeds. Score shifts generalize across families; paired intervals and frozen readouts identify the accompanying discrimination changes.
Synthetic step labels: an exploratory test of the suggestion
Following the ScalePRM suggestion [8], we use existing GenPRM-7B, R-PRM-7B, and Qwen3.5-9B judge outputs for all 25,697 steps of 3,400 ProcessBench chains. Each verifier averages correctness probabilities over eight samples per step, thresholds at 0.5, and the three verifiers vote. Only these synthetic labels select one threshold per frozen PRM; gold labels evaluate the selected rule.
Synthetic calibration recovers about 99% of the summed Fixed–Best gaps, ending 0.27 points below mean Best. Six models improve over Fixed; already-aligned Qwen-Math changes by −0.28. This is exploratory, transductive calibration on the same target chains, without gold-label threshold tuning. Frozen scores are unchanged: the recovered performance comes from the boundary. We will cite ScalePRM and report all model results, label construction, and verification costs.
[8] ScalePRM, arXiv:2512.03244.