- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper asks whether PRMs score equally valid reasoning differently depending on which model wrote it. Comparing verified-correct solutions to the same problem, one from the PRM's own base model and one from another model, it finds that most of the eight PRMs studied give higher scores to their own base model's trajectories on most of six math benchmarks. Two PRMs (Granite-Math and ReasonEval) show weak or reversed preference. The proposed SPARC trains a LoRA adapter on a frozen PRM with three terms: a loss that pushes the mean own-base minus other-model score gap toward zero, a loss on the change in that gap between adjacent progress points, and a terminal protection loss that keeps correct solutions ranked above incorrect ones under a capped target margin. On Qwen-Math, RLHFlow-LLaMA and Skywork PRMs, evaluated on held-out splits of GSM1k, MATH-P and LiveAoPS, SPARC reduces the deviation of the gap from parity by 72.8% on average while raising a correct-versus-incorrect ranking accuracy, and gives small changes on ProcessBench, PRMBench and AceMath-RewardBench selection.
Critical strengths
The diagnostic question is useful and the measurement is careful. PRM scores are increasingly used to compare solutions from different generators, and the paper controls for the obvious confound by comparing only verified-correct solutions to the same problem, then separately shows that unconditioned self-preference rates track base-model accuracy and audits incorrect own-base versus correct other-model pairs. The study covers eight PRMs and six benchmarks with problem-bootstrap intervals, and reports the cases that do not fit the story (Granite-Math below parity in fifteen of sixteen cells, ReasonEval changing direction) instead of dropping them. The progress-resolved view, showing the preference already at 25% of the reasoning, is new for PRMs.
The experimental reporting is transparent. Hyperparameters are chosen on a development split with the selection sequence and rule shown, three seeds are averaged, the paper discloses that the selection rule was revised and that the LiveAoPS test window overlaps earlier development views, and code is released.
Critical weaknesses
The main concern is that the paper does not show that removing the source gap makes the PRM better at its job, which is its core claim. The calibration target assumes the gap is pure bias, and that assumption is not tested. The measured benefits are also explained by the correctness-protection term rather than by calibration.
The target may be wrong. The paper does not show that the measured gap is bias rather than a real quality difference. Matching on final-answer correctness does not match intermediate quality: two correct solutions can differ in step validity, redundancy and how much of the solution is done at the same relative progress. Several of these PRMs were trained on labels from their own family's generations, so "own-base preference" is also consistent with familiarity with the training distribution, and the analysis cannot separate the base model from the training-data generator (Qwen-Math and Skywork share the same base model). Prior work already reports that PRMs score differently across generators because of reasoning-pattern shift (RetrievalPRM, Zhu et al., ACL 2025, https://arxiv.org/abs/2502.14361), which is not cited. This matters for the method: SPARC pushes the mean correct-correct gap to zero, which is the right target only if the two sources are equally good at every progress point. No step-level check (human step labels, ProcessBench-style annotation or a judge on the matched prefixes) is questioning this assumption, so part of what SPARC removes may be signal.
The evidence that calibration improves the PRM is weak. The ranking accuracy averages three conditions, one of which (other-model correct above own-base incorrect) is exactly the comparison that own-base preference hurts, so reducing the gap raises this condition almost by construction. The gains also come from the protection loss, not from calibration: training with the protection loss alone gives nearly the same accuracy as Full SPARC for all three PRMs, and BT-LoRA, which only trains on correct/incorrect pairs, beats SPARC on accuracy in two LLaMA blocks. The downstream results compare only the original PRM with SPARC, the ProcessBench and PRMBench changes are mostly under one point with some decreases (Qwen PRMBench 65.45 to 64.60), and the AceMath-RewardBench numbers are summarized as "12 of 15 improve" without magnitudes or intervals. The Section 4.3 claim that "self-preference calibration can improve PRM performance" therefore needs a protection-only or BT-LoRA baseline on the same downstream benchmarks.
Questions for authors
Can you check whether matched own-base and other-model correct trajectories differ in step quality, for example with human step labels or an LLM judge on the 25/50/75% prefixes? If own-base trajectories have fewer invalid or redundant steps, how much of the gap remains after controlling for this?
Do the RLHFlow PRMs trained on Mistral or DeepSeek data prefer Mistral or DeepSeek generations over Llama ones? This would separate base-model identity from training-data familiarity. Please also list the seven to nine generators used and which of them share a family with each PRM's training data.
How do the protection-only variant and BT-LoRA perform on ProcessBench, PRMBench and AceMath-RewardBench?
Can you report the source-agnostic ranking condition separately in the main table, and a paired win rate for correct-correct pairs alongside GAP?
Given that removing the drift loss leaves the progress-resolved gaps as close to parity or closer, what does the adjacent-step loss contribute?
In which practical setting does source bias change outcomes? An evaluation of best-of-N selection on candidate pools that mix own-base and other-model solutions would show whether SPARC helps where the bias should matter most.
3 comments
Q3: How do Protection-only and BT-LoRA perform on ProcessBench, PRMBench and AceMath-RewardBench?
ProcessBench and PRMBench. New supplementary rows are three-seed means.
| PRM | Method | GSM8K F1 | MATH F1 | Olympiad F1 | OmniMath F1 | PRMBench |
|---|---|---|---|---|---|---|
| LLaMA | Original PRM | 52.72 | 34.41 | 14.72 | 16.61 | 54.57 |
| LLaMA | Protection-only | 51.53 | 34.22 | 14.29 | 16.76 | 55.00 |
| LLaMA | BT-LoRA | 45.85 | 23.19 | 8.59 | 12.17 | 55.61 |
| LLaMA | Full SPARC | 52.89 | 35.26 | 14.83 | 16.90 | 54.97 |
Full SPARC leads on all four ProcessBench subsets, close on PRMBench.
AceMath-RewardBench. LLaMA rm@8 accuracy:
| Subset | Original | Full SPARC |
|---|---|---|
| GSM8K | 93.81 | 93.97 |
| MATH500 | 72.23 | 73.90 |
| Minerva Math | 34.76 | 35.67 |
| Gaokao 2023 EN | 60.27 | 61.43 |
| OlympiadBench | 34.42 | 35.96 |
| College Math | 42.40 | 43.23 |
| MMLU-STEM | 64.09 | 64.84 |
Calibration beyond correctness supervision. Wrong-own/correct-other pairs:
| Method | Seed 17 | Seed 29 | Seed 43 | Mean accuracy |
|---|---|---|---|---|
| Original PRM | 56 | 60 | 57 | 36.27% |
| BT-LoRA | 67 | 71 | 69 | 43.40% |
| Protection-only | 85 | 84 | 82 | 52.62% |
| Full SPARC | 94 | 90 | 88 | 57.02% |
Full SPARC is 13.63pp above BT-LoRA.
Q4: Can you report the source-agnostic condition G separately, and a paired win rate for correct–correct pairs alongside GAP?
Condition G. Terminal accuracy (%) on source-unrestricted pairs (Table 9):
| PRM | Original PRM | Full SPARC |
|---|---|---|
| Qwen | 89.9 | 91.4 |
| LLaMA | 66.6 | 81.6 |
| Skywork | 92.4 | 91.9 |
Condition B is what the target repairs; A blocks uniform lowering.
Paired win rate. Scale-invariant, no condition B:
| Correct–correct population | Original PRM | BT-LoRA | Protection-only | Full SPARC |
|---|---|---|---|---|
| Added matched-control analysis (3,022 relations, 410 problems) | 81.31% | 81.89% | 73.26% | 57.74% |
In the matched-control analysis, Full SPARC lowers by 15.51pp [−17.72, −13.31] relative to Protection-only and by 24.14pp [−27.04, −21.31] relative to BT-LoRA. BT-LoRA leaves at the Original level.
Q5: Given that removing the drift loss leaves the progress-resolved gaps as close to parity or closer, what does the adjacent-step loss contribute?
Drift constrains adjacent-gap changes (Table 9):
| PRM | GAP 75% → 100%, w/o drift | GAP 75% → 100%, Full SPARC |
|---|---|---|
| Qwen | 1.30 → 1.63 | 1.56 → 1.58 |
| LLaMA | 1.34 → 1.43 | 1.25 → 1.16 |
| Skywork | 1.40 → 1.58 | 1.45 → 1.57 |
Without drift the gap reopens toward completion; with it the final-step gap is lower, and Qwen terminal RMS falls 0.651 → 0.567.
Q6: In which practical setting does source bias change outcomes, e.g., best-of-N on pools that mix own-base and other-model solutions?
Source bias matters when a mixed pool holds an incorrect own-base answer and a correct answer from another model—the mixed best-of-2 decision. Formally, for one problem let be the set of correct other-model answers; a violation occurs when , and the violation rate over problems where such a pair exists is .
Violation rates (Figure 8):
| PRM | GSM8K | MATH500 |
|---|---|---|
| Qwen-Math | 66.7% | 52.6% |
| Skywork | 63.6% | 47.4% |
| RLHFlow Mistral-data | 86.1% | 69.0% |
| RLHFlow DeepSeek-data | 84.9% | 55.9% |
On such held-out pairs, the correct answer is selected in:
| Method | Mean accuracy (seeds 17/29/43) |
|---|---|
| Original PRM | 36.27% |
| BT-LoRA | 43.40% |
| Protection-only | 52.62% |
| Full SPARC | 57.02% |
Calibration raises mixed-pool accuracy by 20.75pp over the Original PRM and 4.40pp over the same terminal protection.
Dear Reviewer,
Hope you’re doing well. The discussion deadline is approaching, and we would appreciate it if you could share your feedback at your earliest convenience. We are eager to address your concerns thoroughly and hope we can earn a favorable evaluation.
Best regards
We thank the reviewer. This reply covers Weakness 1–2 and Q1–Q2; the second covers Q3–Q6.
Weakness 1: The target may be wrong — whether the measured gap is bias or a real quality difference, and relation to RetrievalPRM
We will cite RetrievalPRM (Zhu et al., Findings of ACL 2025), which shows PRMs generalize poorly to unfamiliar reasoning; ours asks a complementary question: among verified-correct solutions, does the score depend on which model wrote it, and how does this change across progress?
Whether the gap is bias or quality is answered in Q1; the base-model versus training-data question is answered in Q2.
Weakness 2: The evidence that calibration improves the PRM is weak, and the gains may come from the protection loss
Protection-only keeps Acc high but leaves the preference: Qwen GAP 2.75–4.12× vs. 1.30–1.58× for Full SPARC (Table 9).
BT-LoRA illustrates the difference between correctness training and calibration:
Correctness training raises Acc but can enlarge the gap; SPARC does not.
Q1: Do matched prefixes differ in step quality, and how much of the gap remains?
We added a blinded audit of 1,952 prefixes (488 per anchor) from verified-correct own-base and other-model trajectories. DeepSeek-V4.1-Flash judged each prefix in isolation, without source identity, paired prefix, later steps, reference answer or PRM score.
The sources are at the same stage at every anchor (≤ 0.29).
Matched pairs (validity, progress, redundancy; 448 pairs, 195 problems):
How much remains: the Original PRM still prefers its own base in 84.41% of pairs; SPARC reaches 57.04%, 19.57pp [−22.40, −16.72] below Protection-only.
Equation (1) penalizes (E[g])2 rather than E[g2]=(E[g])2+Var(g), so individual pairs keep quality-driven differences and only the systematic offset is calibrated:
Q2: Do the RLHFlow PRMs prefer generations matching their training data?
Base model versus training data. Two pairs of PRMs separate the two factors. GAP is own-base correct versus other-model correct (Table 4; 1× = no preference):
Base identity is the consistent driver; separating it fully from training data needs controlled provenance, left to future work.