acceptodds
Under review as a conference paper at ICLR 2027

SPARC: Calibrating Self-Preference Across Reasoning Progress in Process Reward Models

Abstract

Process reward models (PRMs) are increasingly used to evaluate intermediate reasoning steps and guide inference-time computation in large language models. Yet it remains unclear whether PRMs score equally valid reasoning differently depending on its source, particularly whether they favor trajectories generated by their own base models. Our analysis reveals two key properties of self-preference in PRMs. First, even among verified-correct solutions to the same problem, PRMs often assign higher scores to trajectories generated by their own base models. Second, self-preference is already present at intermediate reasoning steps and often becomes stronger over the intermediate stages of reasoning. Motivated by these findings, we introduce Self Preference Aware Reward Calibration (SPARC), a process-aware method that calibrates source-dependent score differences across reasoning progress while preserving the PRM’s ability to distinguish correct from incorrect solutions. Across multiple PRMs and mathematical reasoning benchmarks, SPARC reduces source-dependent score deviations from parity by 72.8% on average across reasoning progress while preserving correct–incorrect discrimination. Code is available at https://anonymous.4open.science/r/SPARC-8205.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Process Reward Model, Self-Preference, Reward Calibration, Step-Level Analysis, Mathematical Reasoning

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 4 reviews, above 27% of the others.
Reviewer_QvznOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper asks whether PRMs score equally valid reasoning differently depending on which model wrote it. Comparing verified-correct solutions to the same problem, one from the PRM's own base model and one from another model, it finds that most of the eight PRMs studied give higher scores to their own base model's trajectories on most of six math benchmarks. Two PRMs (Granite-Math and ReasonEval) show weak or reversed preference. The proposed SPARC trains a LoRA adapter on a frozen PRM with three terms: a loss that pushes the mean own-base minus other-model score gap toward zero, a loss on the change in that gap between adjacent progress points, and a terminal protection loss that keeps correct solutions ranked above incorrect ones under a capped target margin. On Qwen-Math, RLHFlow-LLaMA and Skywork PRMs, evaluated on held-out splits of GSM1k, MATH-P and LiveAoPS, SPARC reduces the deviation of the gap from parity by 72.8% on average while raising a correct-versus-incorrect ranking accuracy, and gives small changes on ProcessBench, PRMBench and AceMath-RewardBench selection.

Critical strengths

The diagnostic question is useful and the measurement is careful. PRM scores are increasingly used to compare solutions from different generators, and the paper controls for the obvious confound by comparing only verified-correct solutions to the same problem, then separately shows that unconditioned self-preference rates track base-model accuracy and audits incorrect own-base versus correct other-model pairs. The study covers eight PRMs and six benchmarks with problem-bootstrap intervals, and reports the cases that do not fit the story (Granite-Math below parity in fifteen of sixteen cells, ReasonEval changing direction) instead of dropping them. The progress-resolved view, showing the preference already at 25% of the reasoning, is new for PRMs.

The experimental reporting is transparent. Hyperparameters are chosen on a development split with the selection sequence and rule shown, three seeds are averaged, the paper discloses that the selection rule was revised and that the LiveAoPS test window overlaps earlier development views, and code is released.

Critical weaknesses

The main concern is that the paper does not show that removing the source gap makes the PRM better at its job, which is its core claim. The calibration target assumes the gap is pure bias, and that assumption is not tested. The measured benefits are also explained by the correctness-protection term rather than by calibration.

The target may be wrong. The paper does not show that the measured gap is bias rather than a real quality difference. Matching on final-answer correctness does not match intermediate quality: two correct solutions can differ in step validity, redundancy and how much of the solution is done at the same relative progress. Several of these PRMs were trained on labels from their own family's generations, so "own-base preference" is also consistent with familiarity with the training distribution, and the analysis cannot separate the base model from the training-data generator (Qwen-Math and Skywork share the same base model). Prior work already reports that PRMs score differently across generators because of reasoning-pattern shift (RetrievalPRM, Zhu et al., ACL 2025, https://arxiv.org/abs/2502.14361), which is not cited. This matters for the method: SPARC pushes the mean correct-correct gap to zero, which is the right target only if the two sources are equally good at every progress point. No step-level check (human step labels, ProcessBench-style annotation or a judge on the matched prefixes) is questioning this assumption, so part of what SPARC removes may be signal.

The evidence that calibration improves the PRM is weak. The ranking accuracy averages three conditions, one of which (other-model correct above own-base incorrect) is exactly the comparison that own-base preference hurts, so reducing the gap raises this condition almost by construction. The gains also come from the protection loss, not from calibration: training with the protection loss alone gives nearly the same accuracy as Full SPARC for all three PRMs, and BT-LoRA, which only trains on correct/incorrect pairs, beats SPARC on accuracy in two LLaMA blocks. The downstream results compare only the original PRM with SPARC, the ProcessBench and PRMBench changes are mostly under one point with some decreases (Qwen PRMBench 65.45 to 64.60), and the AceMath-RewardBench numbers are summarized as "12 of 15 improve" without magnitudes or intervals. The Section 4.3 claim that "self-preference calibration can improve PRM performance" therefore needs a protection-only or BT-LoRA baseline on the same downstream benchmarks.

Questions for authors

Can you check whether matched own-base and other-model correct trajectories differ in step quality, for example with human step labels or an LLM judge on the 25/50/75% prefixes? If own-base trajectories have fewer invalid or redundant steps, how much of the gap remains after controlling for this?

Do the RLHFlow PRMs trained on Mistral or DeepSeek data prefer Mistral or DeepSeek generations over Llama ones? This would separate base-model identity from training-data familiarity. Please also list the seven to nine generators used and which of them share a family with each PRM's training data.

How do the protection-only variant and BT-LoRA perform on ProcessBench, PRMBench and AceMath-RewardBench?

Can you report the source-agnostic ranking condition separately in the main table, and a paired win rate for correct-correct pairs alongside GAP?

Given that removing the drift loss leaves the progress-resolved gaps as close to parity or closer, what does the adjacent-step loss contribute?

In which practical setting does source bias change outcomes? An evaluation of best-of-N selection on candidate pools that mix own-base and other-model solutions would show whether SPARC helps where the bias should matter most.

3 comments

We thank the reviewer. This reply covers Weakness 1–2 and Q1–Q2; the second covers Q3–Q6.

Weakness 1: The target may be wrong — whether the measured gap is bias or a real quality difference, and relation to RetrievalPRM

We will cite RetrievalPRM (Zhu et al., Findings of ACL 2025), which shows PRMs generalize poorly to unfamiliar reasoning; ours asks a complementary question: among verified-correct solutions, does the score depend on which model wrote it, and how does this change across progress?

Whether the gap is bias or quality is answered in Q1; the base-model versus training-data question is answered in Q2.

Weakness 2: The evidence that calibration improves the PRM is weak, and the gains may come from the protection loss

Protection-only keeps Acc high but leaves the preference: Qwen GAP 2.75–4.12× vs. 1.30–1.58× for Full SPARC (Table 9).

BT-LoRA illustrates the difference between correctness training and calibration:

LLaMA, Table 1GSM1k: GAP / RMS / AccMATH-P: GAP / RMS / Acc
Original PRM1.44 / 2.215 / 67.66.06 / 2.309 / 65.1
BT-LoRA1.82 / 2.996 / 84.11.60 / 1.811 / 81.4
Full SPARC1.25 / 1.999 / 83.71.13 / 1.028 / 79.4

Correctness training raises Acc but can enlarge the gap; SPARC does not.


Q1: Do matched prefixes differ in step quality, and how much of the gap remains?

We added a blinded audit of 1,952 prefixes (488 per anchor) from verified-correct own-base and other-model trajectories. DeepSeek-V4.1-Flash judged each prefix in isolation, without source identity, paired prefix, later steps, reference answer or PRM score.

Relative step anchorSemantic progress, own / other (0–4)Difference
25%1.12 / 1.15−0.03
50%1.62 / 1.65−0.03
75%2.47 / 2.29+0.18
100%3.42 / 3.14+0.29

The sources are at the same stage at every anchor (≤ 0.29).

Matched pairs (validity, progress, redundancy; 448 pairs, 195 problems):

MethodOwn-source win rate WW on quality-matched pairs
Original PRM84.41%
Protection-only76.61%
Full SPARC57.04%

How much remains: the Original PRM still prefers its own base in 84.41% of pairs; SPARC reaches 57.04%, 19.57pp [−22.40, −16.72] below Protection-only.

Equation (1) penalizes (E[g])2(\mathbb{E}[g])^2 rather than E[g2]=(E[g])2+Var⁡(g)\mathbb{E}[g^2] = (\mathbb{E}[g])^2 + \operatorname{Var}(g), so individual pairs keep quality-driven differences and only the systematic offset is calibrated:

SeedTerminal RMS ↓: group mean / pairwiseTerminal Acc (%) ↑: group mean / pairwise
170.5763 / 0.585577.45 / 76.72
290.5351 / 0.591478.17 / 77.65
430.5792 / 0.666878.01 / 77.30

Q2: Do the RLHFlow PRMs prefer generations matching their training data?

Base model versus training data. Two pairs of PRMs separate the two factors. GAP is own-base correct versus other-model correct (Table 4; 1× = no preference):

PRMBase initializationTraining dataGSM8KMATH500GSM1kMATH-PLiveAoPS
Qwen-MathQwen2.5-Math-7B-InstructQwen pipeline48.86×28.94×7.43×40.20×17.72×
SkyworkQwen2.5-Math-7B-InstructSkywork pipeline5.23×4.54×2.83×2.36×1.71×
RLHFlow Mistral-dataLlama-3.1-8B-InstructMistral generations12.90×4.45×1.67×3.62×2.42×
RLHFlow DeepSeek-dataLlama-3.1-8B-InstructDeepSeek generations19.47×5.55×0.58×7.73×4.35×
  • Same base, different data (Qwen-Math, Skywork). Both favour their shared base on all five benchmarks.
  • Same Llama base, Mistral versus DeepSeek data. Both favour Llama-generated correct trajectories in 9 of 10 cells; switching the data does not remove the preference.

Base identity is the consistent driver; separating it fully from training data needs controlled provenance, left to future work.

Q3: How do Protection-only and BT-LoRA perform on ProcessBench, PRMBench and AceMath-RewardBench?

ProcessBench and PRMBench. New supplementary rows are three-seed means.

PRMMethodGSM8K F1MATH F1Olympiad F1OmniMath F1PRMBench
LLaMAOriginal PRM52.7234.4114.7216.6154.57
LLaMAProtection-only51.5334.2214.2916.7655.00
LLaMABT-LoRA45.8523.198.5912.1755.61
LLaMAFull SPARC52.8935.2614.8316.9054.97

Full SPARC leads on all four ProcessBench subsets, close on PRMBench.

AceMath-RewardBench. LLaMA rm@8 accuracy:

SubsetOriginalFull SPARC
GSM8K93.8193.97
MATH50072.2373.90
Minerva Math34.7635.67
Gaokao 2023 EN60.2761.43
OlympiadBench34.4235.96
College Math42.4043.23
MMLU-STEM64.0964.84

Calibration beyond correctness supervision. Wrong-own/correct-other pairs:

MethodSeed 17Seed 29Seed 43Mean accuracy
Original PRM56605736.27%
BT-LoRA67716943.40%
Protection-only85848252.62%
Full SPARC94908857.02%

Full SPARC is 13.63pp above BT-LoRA.


Q4: Can you report the source-agnostic condition G separately, and a paired win rate for correct–correct pairs alongside GAP?

Condition G. Terminal accuracy (%) on source-unrestricted pairs (Table 9):

PRMOriginal PRMFull SPARC
Qwen89.991.4
LLaMA66.681.6
Skywork92.491.9

Condition B is what the target repairs; A blocks uniform lowering.

Paired win rate. Scale-invariant, no condition B:

Correct–correct populationOriginal PRMBT-LoRAProtection-onlyFull SPARC
Added matched-control analysis (3,022 relations, 410 problems)81.31%81.89%73.26%57.74%

In the matched-control analysis, Full SPARC lowers WW by 15.51pp [−17.72, −13.31] relative to Protection-only and by 24.14pp [−27.04, −21.31] relative to BT-LoRA. BT-LoRA leaves WW at the Original level.


Q5: Given that removing the drift loss leaves the progress-resolved gaps as close to parity or closer, what does the adjacent-step loss contribute?

Drift constrains adjacent-gap changes (Table 9):

PRMGAP 75% → 100%, w/o driftGAP 75% → 100%, Full SPARC
Qwen1.30 → 1.631.56 → 1.58
LLaMA1.34 → 1.431.25 → 1.16
Skywork1.40 → 1.581.45 → 1.57

Without drift the gap reopens toward completion; with it the final-step gap is lower, and Qwen terminal RMS falls 0.651 → 0.567.


Q6: In which practical setting does source bias change outcomes, e.g., best-of-N on pools that mix own-base and other-model solutions?

Source bias matters when a mixed pool holds an incorrect own-base answer and a correct answer from another model—the mixed best-of-2 decision. Formally, for one problem let CC be the set of correct other-model answers; a violation occurs when s(incorrect own-base answer)≥1∣C∣∑c∈Cs(c)s(\text{incorrect own-base answer}) \ge \frac{1}{|C|}\sum_{c \in C} s(c), and the violation rate over problems where such a pair exists is R=violationseligible problems×100%R = \frac{\text{violations}}{\text{eligible problems}} \times 100\%.

Violation rates (Figure 8):

PRMGSM8KMATH500
Qwen-Math66.7%52.6%
Skywork63.6%47.4%
RLHFlow Mistral-data86.1%69.0%
RLHFlow DeepSeek-data84.9%55.9%

On such held-out pairs, the correct answer is selected in:

MethodMean accuracy (seeds 17/29/43)
Original PRM36.27%
BT-LoRA43.40%
Protection-only52.62%
Full SPARC57.02%

Calibration raises mixed-pool accuracy by 20.75pp over the Original PRM and 4.40pp over the same terminal protection.

Dear Reviewer,

Hope you’re doing well. The discussion deadline is approaching, and we would appreciate it if you could share your feedback at your earliest convenience. We are eager to address your concerns thoroughly and hope we can earn a favorable evaluation.

Best regards

Reviewer_n6U7OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper investigates self-preference in process reward models (PRMs), showing that PRMs tend to assign higher scores to reasoning trajectories generated by their own base models, even when compared trajectories are verified correct. The authors further observe that such preference emerges at intermediate reasoning steps and often increases with reasoning progress. To address this, they propose SPARC, a LoRA-based calibration method that minimizes source-dependent score gaps across reasoning progress while preserving correct–incorrect discrimination through a terminal protection loss. Experiments across multiple PRMs and mathematical reasoning benchmarks demonstrate substantial reductions in self-preference, with some improvements in downstream evaluation.

Critical strengths
  1. Interesting and relevant problem. The paper identifies an important but relatively underexplored bias in PRMs. The analysis across reasoning progress provides useful insights beyond existing studies of self-preference in LLM evaluators.

  2. Reasonable method design. SPARC combines progress-aware calibration with correctness protection, explicitly considering the trade-off between reducing source preference and maintaining reward-model discrimination.

  3. Extensive empirical analysis. The experiments cover multiple PRMs, mathematical reasoning benchmarks, competitive baselines, ablations, and generalization settings. The reported reduction in source-dependent score gaps is substantial.

Critical weaknesses
  1. Insufficient isolation of self-preference from reasoning quality. Matching trajectories by final-answer correctness does not guarantee comparable intermediate-step validity, reasoning efficiency, or redundancy. Moreover, aligning trajectories by relative step counts does not necessarily correspond to equivalent semantic progress. Consequently, the observed score gaps may partly reflect genuine differences in reasoning quality or style rather than source preference. Stronger controlled comparisons are needed to establish this distinction.

  2. Limited methodological novelty and unclear necessity of some components. The proposed objective mainly combines group-level score-gap minimization, adjacent-step regularization, and terminal ranking protection. The ablation results (Table 9) show that removing the drift loss often yields comparable performance, while terminal-only calibration sometimes achieves better terminal calibration metrics. This weakens the justification for the additional process-aware components.

  3. Limited and inconsistent downstream improvements. Although SPARC substantially reduces source-dependent gaps, improvements on ProcessBench and PRMBench are often marginal and occasionally negative (Table 2). Candidate-selection improvements are summarized as wins in 12 of 15 settings without sufficiently detailed quantitative results or uncertainty estimates. More importantly, the paper does not demonstrate whether these calibration gains consistently translate into improvements in actual inference-time search or reasoning.

  4. Generalizability and robustness remain uncertain. The calibration experiments are restricted to three PRMs and mathematical reasoning tasks, while some diagnostic models exhibit weak or reversed self-preference (Figure 9). Furthermore, certain generalization experiments involve small evaluation sets, and the held-out-source experiment shows a noticeable accuracy drop on MATH-P (Table 11). These observations limit the strength of the broader generalization claims.

Questions for authors

NA

2 comments

Weakness 1: Insufficient isolation of self-preference from reasoning quality

Blinded audit of 1,952 prefixes (DeepSeek-V4.1-Flash):

Relative step anchorSemantic progress, own / other (0–4)Difference
25%1.12 / 1.15−0.03
50%1.62 / 1.65−0.03
75%2.47 / 2.29+0.18
100%3.42 / 3.14+0.29

Valid-progress-redundancy matched pairs:

MethodWW on quality-matched pairs
Original PRM84.41%
Protection-only (terminal protection, no calibration)76.61%
Full SPARC57.04%

Weakness 2: Limited methodological novelty and unclear necessity of some components

Qwen ablation (Table 9):

VariantGAP 25%GAP 50%GAP 75%GAP 100%RMS ↓Acc (%) ↑
Original PRM6.469.6213.0652.144.33670.71
Full SPARC1.301.441.561.580.56777.88
w/o process loss (Protection-only)2.753.764.124.011.52477.13
w/o protect loss1.011.011.070.890.26768.61
w/o cap2.473.033.826.902.18476.23
w/o step calib. (terminal-only)1.942.492.621.170.40477.87

Drift acts on adjacent gaps:

PRMGAP 75% → 100%, w/o driftGAP 75% → 100%, Full SPARC
Qwen1.30 → 1.631.56 → 1.58
LLaMA1.34 → 1.431.25 → 1.16
Skywork1.40 → 1.581.45 → 1.57

Weakness 3: Limited and inconsistent downstream improvements

Wrong-over-correct ordering:

MethodSeed 17Seed 29Seed 43Mean accuracy
Original PRM56605736.27%
BT-LoRA67716943.40%
Protection-only85848252.62%
Full SPARC94908857.02%

External benchmarks (submitted Original/Full; means for new rows):

MethodGSM8K F1MATH F1Olympiad F1OmniMath F1PRMBench
Original PRM52.7234.4114.7216.6154.57
Protection-only51.5334.2214.2916.7655.00
BT-LoRA45.8523.198.5912.1755.61
Full SPARC52.8935.2614.8316.9054.97

AceMath rm@8 selection accuracy (LLaMA):

SubsetOriginal PRMFull SPARCDifference (pp)
GSM8K93.8193.97+0.16
MATH50072.2373.90+1.66
Minerva Math34.7635.67+0.91
Gaokao 2023 EN60.2761.43+1.17
OlympiadBench34.4235.96+1.54
College Math42.4043.23+0.84
MMLU-STEM64.0964.84+0.75

Best-of-8 selection (200 questions):

Scoring PRMQwen-generated candidatesDeepSeek-generated candidates
Qwen PRM, Original144.5107
Qwen PRM, Full SPARC151.5112
LLaMA PRM, Original145106

Weakness 4: Generalizability and robustness remain uncertain

Own-base preference appears in 6 of the 8 surveyed PRMs (Figure 2, Table 4). Across the calibrated PRMs Full SPARC moves GAP toward parity at every point and cuts mean |log GAP| by 84.1%, 80.7% and 53.7%. Under source hold-out on MATH-P (Table 11), a larger evaluation gives 76.42% (Original) and 74.45% (Full SPARC); excluding MATH-P from training (Table 12) raises Acc from 71.3% to 80.6%.

Dear Reviewer,

Hope you’re doing well. The discussion deadline is approaching, and we would appreciate it if you could share your feedback at your earliest convenience. We are eager to address your concerns thoroughly and hope we can earn a favorable evaluation.

Best regards

Reviewer_S7d4OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper identifies own-base preference in PRMs across reasoning progress and proposes SPARC, a LoRA-based calibration method with correctness protection. Experiments show reduced source-dependent score gaps and mostly improved candidate selection.

Critical strengths
  1. The progress-resolved analysis identifies a potentially important source of bias in reasoning evaluation.
  2. The method explicitly protects correctness discrimination and evaluates practical candidate selection beyond its calibration metric.
Critical weaknesses
  1. The practical necessity of calibration is not fully established. Between-source mean gaps need not harm same-generator ranking, and a controlled comparison against using a different-family PRM is missing.
  2. External evaluations compare the original PRM with full SPARC without correctness-only adaptation baselines. The gains therefore cannot be clearly attributed to debiasing rather than additional correctness supervision.
Questions for authors
  1. Does calibration improve ranking when all candidates come from one generator, and how does it compare with using a different-family PRM?
  2. Can you report BT-LoRA and correctness-only adaptation on the external benchmarks to isolate the contribution of debiasing?
2 comments

We thank the reviewer for the constructive questions. Below, each heading restates one weakness together with the matching question.

Weakness 1 and Q1: The practical necessity of calibration; ranking when all candidates come from one generator; comparison with a different-family PRM

Between-source gaps turn into wrong-over-correct errors

Source preference matters most when the PRM's own base model is wrong and another model is right. For each problem where the base model's answer is incorrect and other models supply at least one correct answer, let CC be the set of correct other-model answers. A violation occurs when s(incorrect own-base answer)≥1∣C∣∑c∈Cs(c)s(\text{incorrect own-base answer}) \ge \frac{1}{|C|}\sum_{c \in C} s(c), and the reported rate is R=violationseligible problems×100%R = \frac{\text{violations}}{\text{eligible problems}} \times 100\%.

PRMGSM8KMATH500
Qwen-Math66.7%52.6%
Skywork63.6%47.4%
RLHFlow Mistral-data86.1%69.0%
RLHFlow DeepSeek-data84.9%55.9%

On these problems an incorrect own-base answer outscores the average correct answer from other models in roughly half to most cases (Figure 8 lists all surveyed PRMs). Calibration repairs this ordering. In an added test on held-out problems where the Qwen PRM's base model is incorrect and another model is correct, we count the pairs ranked correctly:

MethodSeed 17Seed 29Seed 43Mean accuracy
Original PRM56605736.27%
BT-LoRA67716943.40%
Protection-only85848252.62%
Full SPARC94908857.02%

Full SPARC raises this accuracy by 20.75pp over the Original PRM and is the best variant at every seed.

Ranking when all candidates come from one generator, and calibrating versus switching to a different-family PRM

We added a best-of-8 selection test on 200 held-out questions. Each pool contains 8 candidates from a single generator (Qwen2.5-Math-7B-Instruct or DeepSeek-Math-7B-Instruct). We compare calibrating the Qwen PRM with switching to the RLHFlow LLaMA PRM, a different-family PRM. Correctness of the selected candidate is verified against the reference answer.

Scoring PRMQwen-generated candidatesDeepSeek-generated candidates
Qwen PRM, Original144.5107
Qwen PRM, Full SPARC151.5112
LLaMA PRM, Original145106

Each cell counts correct selections out of 200 questions. Calibration improves selection even when all candidates share one generator: +7+7 on Qwen pools and +5+5 on DeepSeek pools. Calibrating the Qwen PRM also selects 6–7 more correct answers than switching to the different-family PRM on both pools.


Weakness 2 and Q2: External evaluations without correctness-only adaptation baselines; BT-LoRA and correctness-only adaptation on the external benchmarks

We report both correctness-only baselines on ProcessBench and PRMBench. Protection-only uses exactly SPARC's terminal protection without the calibration terms, and BT-LoRA trains on correct–incorrect terminal pairs. Original and Full SPARC are the submitted Table 2 values; the new rows are three-seed means.

PRMMethodGSM8K F1MATH F1Olympiad F1OmniMath F1PRMBench
LLaMAOriginal PRM52.7234.4114.7216.6154.57
LLaMAProtection-only51.5334.2214.2916.7655.00
LLaMABT-LoRA45.8523.198.5912.1755.61
LLaMAFull SPARC52.8935.2614.8316.9054.97

Both baselines receive the same or more correctness supervision than SPARC. Full SPARC is nevertheless the best variant on all four ProcessBench subsets and is close to both baselines on PRMBench. The ordering test above isolates the same contribution: with identical terminal protection, adding calibration raises wrong-own/correct-other accuracy from 52.62% to 57.02% (+9/+6/+6 pairs across seeds) and lowers the own-source win rate on correct–correct pairs from 73.26% to 57.74%. These gains therefore come in part from debiasing, alongside the correctness supervision shared by all variants.

Dear Reviewer,

Hope you’re doing well. The discussion deadline is approaching, and we would appreciate it if you could share your feedback at your earliest convenience. We are eager to address your concerns thoroughly and hope we can earn a favorable evaluation.

Best regards

Reviewer_yM4zOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper investigates whether process reward models favor reasoning trajectories generated by their own base models, including at intermediate reasoning stages. It measures source-dependent score gaps between same-problem trajectories with verified correct final answers. SPARC then trains a shared LoRA adapter to reduce group-mean gaps at four progress points and their adjacent changes, while protecting terminal correct–incorrect margins. Experiments report a 72.8% average reduction in absolute log-gap, improved terminal discrimination, and mixed improvements on external process-evaluation benchmarks.

Critical strengths
  1. Studying source dependence within the reasoning process is worthwhile. The analysis goes beyond complete-answer evaluation and documents substantial variation across PRMs and reasoning stages.
  2. The evaluation includes three training seeds, component ablations, and excluded-source and excluded-family experiments. The paper also reports cases where improved source parity does not improve reasoning evaluation.
Critical weaknesses
  1. The calibration target is not sufficiently separated from differences in reasoning quality.

Matching the problem and final-answer correctness is a useful control, but correct solutions can still differ in intermediate validity and how much reasoning has been completed at the same relative step count. Equation (1) penalizes group means rather than individual pairwise differences; nevertheless, systematic quality differences across sources could produce a legitimate group-level score gap. The paper acknowledges this limitation, but the objective still encourages these gaps to vanish. Terminal correctness protection does not directly preserve intermediate quality distinctions. A controlled subset with independently assessed step quality or more closely matched reasoning prefixes would help establish that SPARC removes undesirable source preference.

  1. The evaluation does not establish how much improvement comes from correcting source preference.

Under positive rescaling (s′=as)(s'=as), both (∣log⁡GAP∣)(|\log \mathrm{GAP}|) and group RMS decrease by a factor of (a), while pairwise rankings remain unchanged. Setting (a=0.272) therefore reproduces the headline 72.8% reduction without changing source win rates. SPARC’s Acc improvements show that its overall effect is not merely uniform rescaling, but a rescaling baseline and scale-invariant preference measurements are still needed to interpret the gap reduction. Moreover, the protection-only variant already achieves accuracy close to Full SPARC in Table 9, whereas downstream evaluations compare only the original PRM with Full SPARC. Including this existing control would clarify what process-aware calibration adds beyond terminal correctness training.

Questions for authors

See Critical Weaknesses above.

2 comments

We thank the reviewer. Each heading restates one weakness.

Weakness 1: The calibration target is not sufficiently separated from differences in reasoning quality

We added a blinded audit of 1,952 prefixes (488 per anchor) from verified-correct own-base and other-model trajectories with DeepSeek-V4.1-Flash, blinded to source, pair, later steps, reference answer and score, labelling validity, redundancy and progress (0–4).

Relative step anchorSemantic progress, own / other (0–4)Difference
25%1.12 / 1.15−0.03
50%1.62 / 1.65−0.03
75%2.47 / 2.29+0.18
100%3.42 / 3.14+0.29

Progress rises with the anchor in both sources, and the gap never exceeds 0.29 points. Restricting to pairs where both prefixes are valid and share progress level and redundancy level (448 pairs, 195 problems), the own-source win rate W is:

MethodWW on quality-matched pairs
Original PRM84.41%
Protection-only (terminal protection, no calibration)76.61%
Full SPARC57.04%

The preference survives quality matching: −27.37pp relative to the Original PRM and −19.57pp [−22.40, −16.72] relative to the same terminal protection.

Equation (1) penalizes (E[g])² rather than E[g²], so individual pairs keep quality-driven differences; the group-mean target is also better behaved:

SeedTerminal RMS ↓: group mean / pairwiseTerminal Acc (%) ↑: group mean / pairwise
170.5763 / 0.585577.45 / 76.72
290.5351 / 0.591478.17 / 77.65
430.5792 / 0.666878.01 / 77.30

Weakness 2: The evaluation does not establish how much improvement comes from correcting source preference

A positive rescaling scales |log GAP| and RMS by τ but preserves signs, leaving Acc and W unchanged; rescaling Original by τ = 0.272 reproduces the 72.8% reduction with both unchanged. SPARC moves both: mean |log GAP| falls 84.1%, Acc rises 70.71→77.88%, W falls 84.41→57.04% (quality-matched) and 81.31→57.74% (matched control, 3,022 relations, 410 problems).

VariantGAP 25%GAP 50%GAP 75%GAP 100%RMS ↓Acc (%) ↑
Original PRM6.469.6213.0652.144.33670.71
Protection-only (w/o process loss)2.753.764.124.011.52477.13
w/o protect loss1.011.011.070.890.26768.61
Full SPARC1.301.441.561.580.56777.88

Terminal protection keeps discrimination but leaves the preference; calibration removes it at the highest Acc. It also repairs wrong-over-correct ordering:

MethodSeed 17Seed 29Seed 43Mean accuracyvs. Protection-only (17 / 29 / 43)
Original PRM56605736.27%−29 / −24 / −25
BT-LoRA67716943.40%−18 / −13 / −13
Protection-only85848252.62%0 / 0 / 0
Full SPARC94908857.02%+9 / +6 / +6

Calibration corrects +9/+6/+6 pairs across seeds. External benchmarks (submitted values for Original and Full; means for new rows):

MethodGSM8K F1MATH F1Olympiad F1OmniMath F1PRMBench
Original PRM52.7234.4114.7216.6154.57
Protection-only51.5334.2214.2916.7655.00
BT-LoRA45.8523.198.5912.1755.61
Full SPARC52.8935.2614.8316.9054.97

Full SPARC is the best variant on all four ProcessBench subsets and close to both baselines on PRMBench.

Dear Reviewer,

Hope you’re doing well. The discussion deadline is approaching, and we would appreciate it if you could share your feedback at your earliest convenience. We are eager to address your concerns thoroughly and hope we can earn a favorable evaluation.

Best regards

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.