acceptodds
Desk rejected at ICLR 2027

CM-PRM: Controlled-Margin Contrastive Learning for Process Reward Models

Desk rejected at ICLR 2027, so left out of the paper lists and the map.

Abstract

Process reward models (PRMs) provide fine-grained step-level supervision for complex reasoning tasks. Existing discriminative PRMs offer strong step-level discrimination but rely on large-scale step-level annotations, whereas generative PRMs are more annotation-efficient and interpretable but often lose discriminative capability when trained with existing preference optimization objectives. To bridge this gap, we study generative PRM training from a conditional mutual information perspective and formulate an optimization objective by maximizing a variational lower bound to capture the dependence between process states and generated evaluations. From this perspective, we identify two potential failure modes of these objectives: Synchronous Collapse, where margin-driven optimization allows both chosen and rejected rewards to decrease as their margin grows, weakening process-level discrimination; and Log-ratio Inflatio}, where insufficient control over log-ratio scales can over-reinforce particular evaluation trajectories, encouraging overreliance on local analysis at the expense of other relevant evidence. To address these failure modes, we propose CM-PRM, a controlled-margin contrastive learning framework that decouples the optimization signals for correct and incorrect evaluation trajectories through a Pearson- variational objective and introduces quadratic regularization to control log-ratio scales and stabilize reward outputs. Experiments on standard benchmarks and test-time scaling tasks show that CM-PRM outperforms compared PRMs overall; ablation studies and training-dynamics analyses show that its joint design mitigates simultaneous decline of chosen and rejected rewards and curbs excessive growth in log-ratio magnitudes.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
Reinforcement Learning, LLMs, Process Reward Model

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_z5CaOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper targets generative PRMs and argues that DPO-style training of such models has two failure modes: In "Synchronous Collapse", the log-ratios of the chosen and rejected trajectories both fall while the margin between them grows, which the authors link to verdicts that accept steps without checking them. In "Log-ratio Inflation", the size of the log-ratios is left uncontrolled, which the authors link to verdicts that over-rely on one local observation. Due to these observations, the authors propose CM-PRM, which reformulates DPO as a two-sample InfoNCE bound, then replace the KL divergence with a Pearson chi-squared bound. They then add a separate quadratic penalty on each log-ratio. In the experiments, they use Qwen2.5-Math-7B-Instruct with LoRA and R-PRM data: CM-PRM reaches higher F1 on PRMBench, ProcessBench and higher average accuracy in PRM-guided greedy search over six math sets.

Critical strengths
  1. The paper provides a clear diagnosis of how DPO behaves when training generative PRMs. The paper names a concrete failure of margin-only objectives in this setting and shows it in the training curves. The falling chosen likelihood has been reported for general DPO before, but showing it in this setting, and showing that a growing margin can fix it, is useful to researchers.
  2. CM-PRM gets consistent gains on all three evaluations and the ablations explain where the gain comes from. Every variant starts from the same SFT checkpoint and uses the same data and training budget, so the comparison is fair. CM-PRM beats SFT+DPO on PRMBench, ProcessBench and guided search, and Figures 5 and 6 match the stated mechanism.
Critical weaknesses
  1. The loss terms need clarification and missing baselines. DPO loss depends on the difference between the chosen and rejected log-ratios, in my understanding, Eq. 11 expands into a sum of two independent terms, one involving only the chosen log-ratio and one involving only the rejected log-ratio. Each term is a quadratic with its minimum at a fixed value, so the loss simply pulls the chosen log-ratio toward a positive target and the rejected log-ratio toward a negative target, and the "controlled margin" is just the fixed gap between these two targets. The loss never compares the two trajectories directly, they interact only through the shared model parameters. This is close to existing fixes for the same problem. Cal-DPO (Xiao et al., 2024) pulls the chosen and rejected rewards toward fixed targets of opposite sign, NCA (Chen et al., 2024) optimizes each response's absolute likelihood, and DPOP (Pal et al., 2024) is cited but not run. None of these is compared in the results section. Although these methods were designed for general alignment rather than PRMs, each of them is a drop-in replacement for the DPO loss using the same chosen and rejected log-ratios, so they can be run in the paper's second stage without other changes.

Xiao et al. Cal-DPO: Calibrated Direct Preference Optimization for Language Model Alignment. NeurIPS 2024. https://arxiv.org/abs/2412.14516 Chen et al. Noise Contrastive Alignment of Language Models with Explicit Rewards. NeurIPS 2024. https://arxiv.org/abs/2402.05369 Pal et al. Smaug: Fixing Failure Modes of Preference Optimisation with DPO-Positive. 2024. https://arxiv.org/abs/2402.13228

  1. The experimental setups and results need more clarifications, e.g.: Part of the ProcessBench gain looks like a threshold shift that CM-PRM catches more errors but accepts fewer correct steps, on GSM8K and MATH. The reproduced DPO, trained on a 128K subset whose selection is not described, is weaker than the published R-PRM model. Everything uses one backbone with LoRA, one dataset (R-PRM) and one search setup, so transfer is untested. The link between log-ratio inflation and over-reliance on local analysis is shown on two illustrative examples, not attributed to any specific trained model, with no frequency comparison between DPO and CM-PRM.
Questions for authors
  1. How does CM-PRM compare with Cal-DPO, NCA and DPOP under the same SFT checkpoint and data? If they match CM-PRM, the contribution shrinks to an application study.
  2. Since verdict correctness is a verifiable reward, why is offline preference optimization the right second-stage objective, rather than SFT on correct-verdict trajectories or online RL such as GRPO on the same data? A comparison with an RL-trained generative PRM under the same backbone and data would show whether fixing DPO is the best use of this training budget.
  3. How much of the ProcessBench gain remains if each method's verdict threshold (the probability of "Yes" needed to accept a step) is tuned on validation data? CM-PRM detects more errors but accepts fewer correct steps than DPO, which a threshold shift alone could partly produce.
  4. In section 4.1, how was the 128K preference subset chosen from R-PRM's 269K preference samples? How do DPO and CM-PRM compare when both are trained on the full data?
  5. Can you report CM-PRM vs. SFT+DPO on at least one other backbone (for example, another Qwen model or a Llama model)?
  6. Can you give direct evidence that CM-PRM reduces "over-reliance on local analysis", for example a manually annotated error-type breakdown on a sample of judgments from DPO and CM-PRM?
5 comments
AuthorsOpenReview ↗

Response to Reviewer z5Ca (Part1)

We thank the reviewer for the careful review and for recognizing our analysis of the problem, empirical gains, and controlled ablations. We address your concerns point by point below.

MethodPRMBench-SimplicityPRMBench-SoundnessPRMBench-SensitivityPRMBench-Average
Cal-DPO48.666.172.561.1
NCA51.165.668.961.2
DPOP50.965.573.161.7
DPO-full49.665.572.661.1
CM-PRM-full50.665.873.261.8
Qwen-Math-1.5B-DPO46.657.869.255.3
Qwen-Math-1.5B-CM47.958.570.156.2
CM-PRM (SFT+SFT)51.261.870.959.3
MethodProcessBench-GSM8KProcessBench-MATHProcessBench-OlympiadBenchProcessBench-OmniMathProcessBench-Average
Cal-DPO0.7720.6600.6570.5490.660
NCA0.8010.6740.5850.4760.634
DPOP0.8060.6680.5200.5050.625
DPO-full0.8040.6740.6040.5240.652
CM-PRM-full0.8190.6960.6190.5810.679
Qwen-Math-1.5B-DPO0.5460.5450.3870.4110.473
Qwen-Math-1.5B-CM0.6210.5880.3220.4440.494
CM-PRM (SFT+SFT)0.7820.6440.5540.5280.627

Response to Weakness 1 / Question 1

We thank the reviewer for suggesting Cal-DPO, NCA, and DPOP as additional baselines. We further examined these works and their training mechanisms, and acknowledge that their objectives, although not originally developed for PRMs, are applicable to PRM preference learning. However, applicability does not imply identical optimization constraints or effectiveness. We therefore added objective-level analyses and direct comparisons under matched conditions, which show that CM’s objective configuration is more effective in our mathematical process-evaluation setting.

  • CM-PRM vs. Cal-DPO / DPOP

Let Δ=hw−hl\Delta=h_w-h_l and [z]+=max⁡(z,0)[z]_+=\max(z,0). The two losses are

=-\log\sigma(\Delta)+(h_w-a)^2+(h_l+a)^2, \qquad \ell_{\mathrm{DPOP}} =-\log\sigma\left(b[\Delta-\lambda[-h_w]_+]\right).$$ Cal-DPO retains the Bradley-Terry (BT) pairwise ranking term, whereas DPOP adds a one-sided penalty when the chosen likelihood falls below the reference likelihood. Both retain gradient-coefficient dependence on the relative margin, preserving the coupling identified in our analysis of Challenge 1: Synchronous Collapse. Moreover, DPOP’s additional penalty is inactive when the chosen likelihood reaches or exceeds the reference level, leaving no continuous individual control over both absolute scales. In contrast, each CM coefficient responds separately to its own log-ratio, so reinforcement of correct-verdict trajectories is no longer jointly determined by the relative gap between the two trajectories. - **CM-PRM vs. NCA** The pairwise-preference version of NCA uses $$\ell_{\mathrm{NCA}} =-\log\sigma(r_w)-\tfrac12\log\sigma(-r_w)-\tfrac12\log\sigma(-r_l), \qquad \frac{\partial\ell_{\mathrm{NCA}}}{\partial r_l} =\frac{1}{2}\sigma(r_l)>0,$$ where $r_i$ denotes a scaled log-ratio. Along the rejected coordinate, decreasing $r_l$ always reduces the loss. There is no finite negative target or corrective feedback that reverses excessive suppression, leaving incomplete two-sided scale control against Challenge 2: Log-ratio Inflation. CM instead sets finite targets on both sides and reverses the corresponding feedback after a target is crossed. This provides reinforcement signals for learning correct-verdict trajectories while limiting excessive suppression of incorrect-verdict trajectories, rather than primarily expanding the margin by lowering rejected likelihoods. - **Empirical Analysis** We added controlled comparisons with all three baselines. All methods share the same SFT checkpoint, reference model, exactly the same 128K preference examples, and training/validation splits. Except for the objective and its specific hyperparameters, the training budget, learning rate, batch size, LoRA configuration, and evaluation settings are identical. Objective-specific hyperparameters and checkpoints are selected using the same validation set and F1 criterion, without test-set tuning. CM-PRM achieves 69.1 on PRMBench, outperforming Cal-DPO, NCA, and DPOP by 13.1%, 12.9%, and 12.0%, respectively. On ProcessBench, it achieves 71.3, with corresponding relative improvements of 8.0%, 12.5%, and 14.1%. These results demonstrate that, under the same supervision and training budget, CM-PRM is more effective than these three objectives. Its joint control design provides practical benefits rather than being an equivalent substitution for an existing absolute log-ratio objective.
AuthorsOpenReview ↗

Response to Reviewer z5Ca (Part2)

Note: This is a continuation of our Response to Weakness 1 / Question 1.

  • Rationale for Trajectory-Wise Control

Regarding Eq. (11), completing the square does not undermine its design rationale; rather, it explicitly reveals the structure of trajectory-wise control. For margin-only objectives, jointly shifting hwh_w and hlh_l leaves the loss unchanged, so their common drift remains unconstrained. CM instead controls their absolute positions separately. Thus, the “controlled margin” does not merely specify a fixed gap between two targets; it separately regulates reinforcement and suppression scales through trajectory-wise feedback.

Response to Question 2

Regarding continued SFT, we added a second-stage control trained only on correct-verdict trajectories, denoted CM-PRM (SFT+SFT). Starting from the same SFT checkpoint and matching the second-stage training budget, this control achieves 59.3 on PRMBench and 62.7 on ProcessBench, compared with CM-PRM’s 69.1 and 71.3, yielding relative improvements of 16.5% and 13.7%. Under this setting, simply extending SFT on correct-verdict trajectories is insufficient to achieve comparable performance; using optimization signals from both correct and incorrect trajectories is more effective.

Regarding GRPO, we consider three aspects.

  1. A verifiable reward does not make GRPO a direct replacement for our optimization design. Online learning depends on the current policy sampling informative trajectories. When correct evaluations are difficult for the model to generate, obtaining useful positive signals requires additional exploration, whereas existing offline data can provide this supervision directly. Standard GRPO also relies on within-group relative rewards. When every trajectory in a group receives the same correctness reward, reward differences cannot provide a discriminative signal, whereas offline preference pairs directly supply contrasts between correct and incorrect verdicts.
  2. GRPO rollouts incur additional computational costs. GRPO requires repeated online generation of multiple analysis and verdict trajectories, followed by reward and group-advantage computation. Consequently, even with the same backbone and prompts, its total cost cannot be equated with offline training solely by matching parameter-update counts; generation costs must also be included.
  3. The practical budget for PRM training must remain bounded. Our goal is to use existing offline supervision to construct an evaluator for subsequent reasoning search and reinforcement learning at limited cost, rather than introducing an additional online RL stage during reward-model construction. Given our available resources, we have not conducted a GRPO comparison.

Our controlled comparisons examine the value of improving the objective under fixed offline data and training budgets; we do not claim that CM outperforms all online training methods.

Response to Weakness 2 / Question 3

ThresholdCM-PRMDPO
0.50.7130.668
0.30.5820.524
0.70.6380.602

Thank you for raising the possibility that a threshold shift could explain part of the improvement. We aim for a PRM that can be applied directly across tasks, rather than requiring threshold selection using validation labels from each new benchmark. Following R-PRM, all methods therefore use the same 0.5 majority-vote threshold, without separate tuning for ProcessBench.

To further examine threshold sensitivity, we added comparisons at thresholds of 0.3, 0.5, and 0.7. CM-PRM outperforms DPO at all three thresholds, with relative improvements of 11.1%, 6.7%, and 6.0%, respectively. Both methods also attain their highest scores among the tested thresholds at 0.5. Thus, CM-PRM’s advantage is not confined to the default threshold; it is reproduced across all three tested settings.

Response to Weakness 2 / Question 4

The 128K preference subset was randomly sampled from R-PRM’s 269K examples. We additionally trained DPO and CM-PRM on the complete 269K preference dataset, starting from the same SFT checkpoint. As shown above, CM-PRM-full achieves 61.8 on PRMBench versus DPO-full’s 61.1 and outperforms DPO-full on all four ProcessBench subsets.

Full-data training does not improve either method’s absolute scores. One possible explanation is distribution mismatch in offline preference optimization: stronger fitting to the offline preference data could exacerbate the mismatch between the current policy and the data-generating policy, potentially impairing generalization. These results do not support the view that adding data necessarily improves performance; meanwhile, CM retains its advantage over DPO under full-data training. We will include the sampling procedure and full-data results in the revision.

AuthorsOpenReview ↗

Response to Reviewer z5Ca (Part3)

Response to Weakness 2 / Question 5

We added a controlled comparison using the Qwen-Math-1.5B backbone. With the same SFT initialization, data, and training settings, CM improves PRMBench performance from DPO’s 55.3 to 56.2, a relative gain of 1.6%, and improves average ProcessBench performance from 47.3 to 49.4, a relative gain of 4.4%. These results provide additional evidence at a model scale beyond the original 7B backbone.

Response to Weakness 2 / Question 6

To test whether models overly rely on familiar local expressions, we conducted a meaning-preserving rewriting experiment on PRMBench’s deception subset. This subset tests whether PRMs can identify reasoning that appears plausible but contains subtle errors, requiring them to distinguish surface form from actual correctness. We selected all positively labeled examples, 7,881 in total, and used DeepSeek-v4 to replace common terms, formulas, and symbols in the problem statements with equivalent but less familiar expressions, preserving mathematical meaning and step correctness. We then compared verdict accuracy on the same examples before and after rewriting. Positive examples were selected to test whether unfamiliar expressions alone would cause a model to incorrectly reject a valid step.

MethodOriginal AccuracyRewritten Accuracy
CM-PRM86.784.9
DPO83.874.5

CM-PRM’s accuracy decreases from 86.7 to 84.9, a relative drop of 2.1%, whereas DPO’s decreases from 83.8 to 74.5, a relative drop of 11.1%. CM-PRM’s smaller degradation indicates that its judgments are less affected by changes in local surface forms. This provides systematic behavioral evidence beyond illustrative cases, supporting its greater robustness to meaning-preserving rewriting.

Reviewer_NsKyOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies preference optimization for generative process reward models that produce an analysis followed by a step-correctness judgment. It argues that relative-margin objectives only constrain the gap between the two evaluations: both likelihoods can drop at the same time — including that of the chosen one — and nothing pins down the absolute scale of each log-ratio. CM-PRM combines a Pearson χ²-inspired objective with separate quadratic penalties on the chosen and rejected log-ratios, pinning each to a finite absolute target rather than only their difference. Using Qwen2.5-Math-7B-Instruct with R-PRM-derived supervision, the paper reports improvements over a DPO baseline trained on identical data from the same backbone, on PRMBench, ProcessBench, and PRM-guided greedy search; objective ablations and training-dynamics analyses are also provided.

Critical strengths
  1. The gains are not limited to static evaluation. Against the matched DPO baseline, CM-PRM improves PRMBench from 65.1 to 69.1, ProcessBench from 66.9 to 71.3, and average greedy-search accuracy from 49.0 to 53.6. The downstream gains matter more here, as they test whether the evaluator can select useful continuations rather than just classify fixed steps.

  2. The ablations are well controlled and the implementation is documented in reasonable detail. All objective variants share the SFT initialization, training data, and configuration, with checkpoints selected on validation. Comparisons cover Pearson alone, both one-sided regularizers, and regularized DPO/IPO; Appendix F further separates repeated inference runs from retraining variability. These controls make the results within this configuration fairly convincing.

Critical weaknesses
  1. The contribution is not clearly separated from closely related absolute log-ratio objectives. Completing the square in Eq. 11 gives separate squared losses centered at 1/(βτ)1/(\beta\tau) and −1/(γτ)-1/(\gamma\tau); with equal coefficients this matches the hard-label pairwise SPPO loss up to positive scaling and a constant. The asymmetric setting and training procedure differ, but the core mechanism is already present in SPPO. Neither SPPO nor Cal-DPO is discussed or evaluated, and the zero-centered DPO+Reg. ablation is not equivalent to Cal-DPO, so the incremental contribution remains unclear.

  2. The evidence supports the predictive improvements more strongly than the mechanistic explanation. The CMI interpretation relies on the stochastic-dominance assumption in Sec. 2.3/App. A.3, not established for the actual training process. The coordinate-wise results do not guarantee probability changes under shared parameters, as App. C.1 acknowledges, and the variance analysis assumes boundedness and estimator regularity. Figs. 5–6 show aggregate optimization behavior, not that log-ratio control reduces insufficient verification or overreliance on local analysis. The cases in App. D only illustrate the failure modes; no experiment measures whether the method reduces them. Since chosen trajectories are selected by final-verdict correctness, better verdict accuracy is not by itself evidence of more reliable analyses.

Questions for authors
  1. How does CM-PRM differ from the hard-label SPPO loss after completing the square? Could you provide matched comparisons with symmetric CM/SPPO and with Cal-DPO? What evidence supports the asymmetric choice over a symmetric finite-target objective?

  2. What supports the assumption that state-specific negative critic scores stochastically dominate conditionally independent ones during training? Is there direct evidence linking log-ratio control to better analysis quality or to the two named failure modes, beyond aggregate curves and final-verdict benchmarks?

  3. Please specify the evaluator trajectory count per step, decoding settings, verdict-to-score aggregation, classification thresholds, and whether log-probabilities are sequence sums or length-normalized. How do the plotted rewards relate to Eq. 11?

2 comments
AuthorsOpenReview ↗

Response to Reviewer NsKy (Part 1)

We thank the reviewer for the careful review and for recognizing the strengths of our experimental design and the performance improvements. We address your concerns point by point below.

MethodPRMBench-SimplicityPRMBench-SoundnessPRMBench-SensitivityPRMBench-Average
Cal-DPO48.666.172.561.1
CM-PRM-shuffle51.664.672.961.3
SPPO47.665.771.360.2
MethodProcessBench-GSM8KProcessBench-MATHProcessBench-OlympiadBenchProcessBench-OmniMathProcessBench-Average
Cal-DPO0.7720.6600.6570.5490.660
SPPO0.7340.6650.5730.5230.624
CM-PRM-shuffle0.7980.6190.5420.5130.618

Response to Weakness 1 / Question 1

Thank you for suggesting comparisons with SPPO and Cal-DPO. We further examined both works and their training mechanisms. Although neither was originally developed for PRM training, we acknowledge that their objectives can be applied to PRM preference learning. However, this applicability does not establish that their constraints are best suited to mathematical process evaluation. We therefore added analytical and empirical comparisons, which demonstrate the advantages of CM’s objective configuration under our experimental setting.

  • CM-PRM vs. SPPO

The key distinction lies in the constraints on target values and feedback curvatures. In common log-ratio coordinates h=(hw,hl)h=(h_w,h_l), the hard-label pairwise SPPO loss is

ℓSPPO=(hw−η/2)2+(hl+η/2)2.\ell_{\mathrm{SPPO}}=(h_w-\eta/2)^2+(h_l+\eta/2)^2.

It imposes symmetric targets ±η/2\pm\eta/2, with second derivatives equal to 2 for both hwh_w and hlh_l. In contrast, CM’s coordinate-wise targets are hw⋆=1/(βτ)h_w^\star=1/(\beta\tau) and hl⋆=−1/(γτ)h_l^\star=-1/(\gamma\tau), with corresponding second derivatives βτ2\beta\tau^2 and γτ2\gamma\tau^2. Both losses have zero mixed second derivatives. CM therefore permits unequal target magnitudes and feedback curvatures, rather than requiring reinforcement and suppression to follow the same symmetric constraint. When β=γ\beta=\gamma and the target parameter is appropriately matched, the two losses indeed differ only by a positive scaling factor and an additive constant. Our configuration, however, is asymmetric. Adjusting SPPO’s η\eta only shifts its symmetric targets, while global loss rescaling changes both curvatures together. Neither adjustment can reproduce CM’s unequal curvatures when β≠γ\beta\ne\gamma.

Figure 5 further shows that different regularization strengths produce different training dynamics and evaluation performance, supporting explicit control over feedback strength. CM can thus separately configure the reinforcement of correct-verdict trajectories and the suppression of incorrect-verdict trajectories, providing incentives to learn correct evaluations while limiting excessive suppression of incorrect trajectories.

  • CM-PRM vs. Cal-DPO

Cal-DPO adds symmetric calibration to the Bradley-Terry (BT) pairwise ranking loss. Let Δ=hw−hl\Delta=h_w-h_l. Its gradients are

\frac{\partial\ell_{\mathrm{Cal}}}{\partial h_l}=2(h_l+a)+\sigma(-\Delta).$$ First, the ranking term makes both feedback coefficients depend on the relative margin, retaining the margin coupling identified in our analysis of Synchronous Collapse. CM’s coefficients instead respond separately to their own log-ratios, providing more direct trajectory-wise scale control. Second, symmetric calibration imposes equal-magnitude targets and identical calibration curvature. When reinforcing correct trajectories and suppressing incorrect trajectories require different scales, a shared parameter imposes a compromise. CM permits separate configurations of the two targets and feedback strengths, without tying reinforcement and suppression to the same symmetric constraint. - **Empirical Analysis** We applied the hard-label SPPO and Cal-DPO losses to the same offline second-stage training setup. All methods share the same SFT checkpoint, reference model, exactly the same 128K preference examples, and training/validation splits; all remaining training and evaluation settings are identical. Both baselines use a target-scale parameter of 0.01, parameterizing their symmetric targets or calibration centers as $\pm1/(2b)$. At a reference length of 1,000 tokens, CM’s configuration corresponds to sequence-level targets of approximately $+66.67$ and $-33.33$, while the baselines use $\pm50$, matching the target gap at that reference length. CM-PRM achieves 69.1 on PRMBench, compared with SPPO’s 60.2 and Cal-DPO’s 61.1, yielding relative improvements of 14.8% and 13.1%, respectively. On ProcessBench, it achieves 71.3 versus 62.4 and 66.0, yielding relative improvements of 14.3% and 8.0%. These results support the practical advantage of CM’s asymmetric target and feedback configuration over the evaluated alternatives.
AuthorsOpenReview ↗

Response to Reviewer NsKy (Part 2)

Response to Weakness 2 / Question 2

CM-PRM-shuffleCM-PRM
Mean rejected critic score−5.01×10−2-5.01\times10^{-2}−6.85×10−5-6.85\times10^{-5}

Regarding the validity of the stochastic-dominance assumption, this assumption underpins the conditional mutual information lower-bound interpretation with hard negatives; it is not required for the CM objective’s gradient-coefficient decoupling or finite coordinate-wise targets. To examine negative construction, we added a shuffle control that randomly reassigns negative evaluation trajectories to states while keeping the other training settings unchanged. CM-PRM improves over the shuffle variant from 61.3 to 69.1 on PRMBench and from 61.8 to 71.3 on ProcessBench, relative gains of 12.7% and 15.4%, respectively. Moreover, the mean critic score for hard negatives is −6.85×10−5-6.85\times10^{-5}, higher than −5.01×10−2-5.01\times10^{-2} for randomly paired negatives. These results demonstrate the practical benefit of hard-negative training, while the mean critic scores are directionally consistent with the assumption Tθ(x,yl)⪰stTθ(x,yliid)T_\theta(x,y_l)\succeq_{\mathrm{st}}T_\theta(x,y_l^{\mathrm{iid}}).

Regarding the mitigation of local-analysis overreliance, we conducted a meaning-preserving rewriting experiment on PRMBench’s deception subset to examine whether CM-PRM’s log-ratio control design alleviates reliance on local expressions. This subset tests whether models can identify reasoning that appears plausible but contains subtle errors, requiring them to distinguish surface form from actual correctness. It is therefore suitable for examining reliance on local surface cues. We selected all positively labeled examples, 7,881 in total, and used DeepSeek-v4 to replace common terms, formulas, and symbols in the problem statements with equivalent but less familiar expressions, preserving mathematical meaning and step correctness. We then compared verdict accuracy on the same examples before and after rewriting. Positive examples were selected to test whether unfamiliar expressions alone would cause a model to incorrectly reject a valid step.

MethodOriginal AccuracyRewritten Accuracy
CM-PRM86.784.9
DPO83.874.5

After rewriting, CM-PRM’s accuracy decreases by only 2.1% relative to its original accuracy, compared with 11.1% for DPO. This shows that, in this test, CM-PRM mitigates evaluation failures induced by changes in local surface forms and is more robust to equivalent but unfamiliar expressions.

Furthermore, PRM-guided greedy search uses generated analyses and verdicts to select continuations step by step, testing whether these outputs help complete reasoning. With the same policy model and search configuration, CM-PRM improves accuracy from DPO’s 49.0 to 53.6 (Table 3 of the main paper), a relative gain of 9.4%. Together, these results indicate that CM-PRM improves the task-relevant quality of its generated analyses and verdicts, contributing to higher final-answer accuracy.

Response to Question 3

Following R-PRM’s evaluation protocol, we generate 10 evaluation trajectories per step, with temperature=0.8, top-p=0.95, and top-k=20. Each trajectory’s Yes/No verdict is mapped to 1/0, and the mean over the 10 trajectories serves as the step score. The classification threshold is fixed at 0.5.

Training log-probabilities are length-normalized, using hˉθ(x,y)=1∣y∣log⁡πθ(y∣x)πref(y∣x)\bar h_\theta(x,y)=\frac{1}{|y|}\log\frac{\pi_\theta(y|x)}{\pi_{\mathrm{ref}}(y|x)}. Equation (11) presents the loss for an individual preference pair; the overall training objective is its expectation over the training-data distribution. We will correct this omission in the revision. The chosen and rejected rewards in Figures 5–6 correspond to hˉw\bar h_w and hˉl\bar h_l, respectively, and the margin is their difference, rather than the proportion of Yes verdicts used at inference. We will clarify their correspondence with Eq. (11) in the figure captions.

Reviewer_6mDUOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

This paper presents CM-PRM, a controlled-margin contrastive learning framework designed to improve the discriminative capability of generative process reward models (PRMs). The authors identify two limitations of conventional preference optimization: Synchronous Collapse, where chosen and rejected rewards decrease simultaneously, and Log-ratio Inflation, where excessive reinforcement may compromise evaluation robustness. To address these issues, CM-PRM combines a contrastive objective derived from Pearson divergence with quadratic regularization to decouple gradient signals and control individual log-ratio scales. Experiments on PRMBench, ProcessBench, and PRM-guided search demonstrate improvements over existing PRMs, supported by ablation studies and training-dynamics analyses.

Critical strengths
  • The proposed objective addresses meaningful limitations of margin-based preference optimization by independently regulating chosen and rejected trajectories. The accompanying gradient analysis and ablation studies provide useful insights into why controlling individual log-ratios can improve process-level discrimination.

  • CM-PRM achieves competitive results on both static process evaluation and PRM-guided search. In particular, it improves PRMBench F1 from 65.1 to 69.1 and guided-search accuracy from 49.0 to 53.6 compared with the matched DPO baseline. These improvements, together with the training-dynamics analyses, provide substantive evidence of the method's effectiveness.

Critical weaknesses
  • The conditional mutual information interpretation depends on a stochastic-dominance assumption regarding hard-negative samples, but the paper does not empirically verify whether this assumption holds. Moreover, the final objective can be expressed as a separable quadratic function of chosen and rejected log-ratios, leaving the necessity of adopting this particular divergence-based formulation insufficiently established. These issues weaken the theoretical justification for the proposed design.

  • The experiments use a single 7B backbone and focus exclusively on mathematical reasoning. Although the paper attributes improvements partly to more reliable and evidence-grounded analyses, the evaluation primarily measures judgment correctness without directly assessing the faithfulness or completeness of generated analyses. Consequently, the generality and underlying mechanism of the improvements remain insufficiently demonstrated.

Questions for authors
  • Can the authors provide empirical evidence supporting the stochastic-dominance assumption used to establish the conditional mutual information lower bound? How sensitive is the theoretical interpretation to violations of this assumption?

  • Since chosen and rejected trajectories are distinguished by final judgment correctness, how does the proposed objective ensure that the generated analyses are themselves logically sound and evidence-grounded? Can the authors provide direct evidence that CM-PRM improves analysis quality rather than merely final judgment accuracy?

  • Can the authors compare CM-PRM with simpler alternatives that independently optimize chosen and rejected trajectories with explicit regularization? Such comparisons would help establish whether the proposed divergence-based formulation provides benefits beyond gradient decoupling and quadratic regularization.

1 comment
AuthorsOpenReview ↗

Response to Reviewer 6mDU

We thank the reviewer for the careful review and for recognizing the strengths of our method design and empirical results. We address your concerns point by point below.

MethodPRMBench-SimplicityPRMBench-SoundnessPRMBench-SensitivityPRMBench-Average
CM-PRM-shuffle51.664.672.961.3
SPPO47.665.771.360.2
MethodProcessBench-GSM8KProcessBench-MATHProcessBench-OlympiadBenchProcessBench-OmniMathProcessBench-Average
CM-PRM-shuffle0.7980.6190.5420.5130.618
SPPO0.7340.6650.5730.5230.624

Response to Weakness 1 / Question 1

CM-PRM-shuffleCM-PRM
Mean rejected critic score−5.01×10−2-5.01\times10^{-2}−6.85×10−5-6.85\times10^{-5}

Regarding the validity of the stochastic-dominance assumption, this assumption underpins the conditional mutual information lower-bound interpretation with hard negatives; it is not required for the CM objective’s gradient-coefficient decoupling or finite coordinate-wise targets. To examine the practical role of negative construction, we added a shuffle control, as reported above. CM-PRM achieves 69.1 on PRMBench versus 61.3 for CM-PRM-shuffle, a relative improvement of 12.7%. On ProcessBench, the corresponding scores are 71.3 and 61.8, a relative improvement of 15.4%. Furthermore, the mean critic score for hard negatives is −6.85×10−5-6.85\times10^{-5}, higher than −5.01×10−2-5.01\times10^{-2} for randomly paired negatives. These results demonstrate the practical benefit of hard-negative training, while the mean critic scores are directionally consistent with the assumption Tθ(x,yl)⪰stTθ(x,yliid)T_\theta(x,y_l)\succeq_{\mathrm{st}}T_\theta(x,y_l^{\mathrm{iid}}).

Regarding the design of our objective function, we aim to jointly achieve gradient-coefficient decoupling and trajectory-wise scale control. Its separable quadratic structure implements this joint design rather than simply combining quadratic penalties. The Pearson objective first removes the gradient coefficients’ dependence on the relative margin, while two-sided regularization separately controls the individual log-ratio scales. In contrast, directly adding quadratic regularization to DPO or IPO retains their original margin coupling. These variants achieve PRMBench scores of 61.1 and 61.7, respectively, below CM-PRM’s 69.1 (Table 2 of the main paper). Thus, the ablations support the joint design of the Pearson objective and individual scale control, rather than merely adding quadratic penalties.

Response to Weakness 2 / Question 2

Our research scope is process evaluation in mathematical reasoning. Within this scope, our primary objective is to improve final-answer accuracy through effective process evaluation, rather than treating the quality of generated analysis text as the primary evaluation endpoint. With the same policy model and search configuration, CM-PRM improves guided greedy-search accuracy from DPO’s 49.0 to 53.6 (Table 3 of the main paper), directly demonstrating its ability to select useful continuations and improve reasoning outcomes. The 7B backbone follows the settings of the major baselines and enables fair, controlled comparisons. We will also explore extensions to other domains, including code generation and scientific reasoning, in future work.

Response to Question 3

To evaluate a simpler independently optimized alternative, we added a hard-label pairwise SPPO baseline [1]. SPPO pulls the chosen and rejected log-ratios toward positive and negative finite targets, directly addressing your suggestion. Under the same initialization, training data, and budget, CM-PRM achieves 69.1 on PRMBench versus SPPO’s 60.2, a relative improvement of 14.8%. On ProcessBench, it achieves 71.3 versus SPPO’s 62.4, a relative improvement of 14.3%. This experiment directly evaluates an alternative that also provides gradient-coefficient decoupling and finite-target control, supporting the practical advantage of CM-PRM’s objective configuration.

[1] Wu, Y., Sun, Z., Yuan, H., et al. Self-Play Preference Optimization for Language Model Alignment. ICLR, 2025.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.