- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper targets generative PRMs and argues that DPO-style training of such models has two failure modes: In "Synchronous Collapse", the log-ratios of the chosen and rejected trajectories both fall while the margin between them grows, which the authors link to verdicts that accept steps without checking them. In "Log-ratio Inflation", the size of the log-ratios is left uncontrolled, which the authors link to verdicts that over-rely on one local observation. Due to these observations, the authors propose CM-PRM, which reformulates DPO as a two-sample InfoNCE bound, then replace the KL divergence with a Pearson chi-squared bound. They then add a separate quadratic penalty on each log-ratio. In the experiments, they use Qwen2.5-Math-7B-Instruct with LoRA and R-PRM data: CM-PRM reaches higher F1 on PRMBench, ProcessBench and higher average accuracy in PRM-guided greedy search over six math sets.
Critical strengths
- The paper provides a clear diagnosis of how DPO behaves when training generative PRMs. The paper names a concrete failure of margin-only objectives in this setting and shows it in the training curves. The falling chosen likelihood has been reported for general DPO before, but showing it in this setting, and showing that a growing margin can fix it, is useful to researchers.
- CM-PRM gets consistent gains on all three evaluations and the ablations explain where the gain comes from. Every variant starts from the same SFT checkpoint and uses the same data and training budget, so the comparison is fair. CM-PRM beats SFT+DPO on PRMBench, ProcessBench and guided search, and Figures 5 and 6 match the stated mechanism.
Critical weaknesses
- The loss terms need clarification and missing baselines. DPO loss depends on the difference between the chosen and rejected log-ratios, in my understanding, Eq. 11 expands into a sum of two independent terms, one involving only the chosen log-ratio and one involving only the rejected log-ratio. Each term is a quadratic with its minimum at a fixed value, so the loss simply pulls the chosen log-ratio toward a positive target and the rejected log-ratio toward a negative target, and the "controlled margin" is just the fixed gap between these two targets. The loss never compares the two trajectories directly, they interact only through the shared model parameters. This is close to existing fixes for the same problem. Cal-DPO (Xiao et al., 2024) pulls the chosen and rejected rewards toward fixed targets of opposite sign, NCA (Chen et al., 2024) optimizes each response's absolute likelihood, and DPOP (Pal et al., 2024) is cited but not run. None of these is compared in the results section. Although these methods were designed for general alignment rather than PRMs, each of them is a drop-in replacement for the DPO loss using the same chosen and rejected log-ratios, so they can be run in the paper's second stage without other changes.
Xiao et al. Cal-DPO: Calibrated Direct Preference Optimization for Language Model Alignment. NeurIPS 2024. https://arxiv.org/abs/2412.14516 Chen et al. Noise Contrastive Alignment of Language Models with Explicit Rewards. NeurIPS 2024. https://arxiv.org/abs/2402.05369 Pal et al. Smaug: Fixing Failure Modes of Preference Optimisation with DPO-Positive. 2024. https://arxiv.org/abs/2402.13228
- The experimental setups and results need more clarifications, e.g.: Part of the ProcessBench gain looks like a threshold shift that CM-PRM catches more errors but accepts fewer correct steps, on GSM8K and MATH. The reproduced DPO, trained on a 128K subset whose selection is not described, is weaker than the published R-PRM model. Everything uses one backbone with LoRA, one dataset (R-PRM) and one search setup, so transfer is untested. The link between log-ratio inflation and over-reliance on local analysis is shown on two illustrative examples, not attributed to any specific trained model, with no frequency comparison between DPO and CM-PRM.
Questions for authors
- How does CM-PRM compare with Cal-DPO, NCA and DPOP under the same SFT checkpoint and data? If they match CM-PRM, the contribution shrinks to an application study.
- Since verdict correctness is a verifiable reward, why is offline preference optimization the right second-stage objective, rather than SFT on correct-verdict trajectories or online RL such as GRPO on the same data? A comparison with an RL-trained generative PRM under the same backbone and data would show whether fixing DPO is the best use of this training budget.
- How much of the ProcessBench gain remains if each method's verdict threshold (the probability of "Yes" needed to accept a step) is tuned on validation data? CM-PRM detects more errors but accepts fewer correct steps than DPO, which a threshold shift alone could partly produce.
- In section 4.1, how was the 128K preference subset chosen from R-PRM's 269K preference samples? How do DPO and CM-PRM compare when both are trained on the full data?
- Can you report CM-PRM vs. SFT+DPO on at least one other backbone (for example, another Qwen model or a Llama model)?
- Can you give direct evidence that CM-PRM reduces "over-reliance on local analysis", for example a manually annotated error-type breakdown on a sample of judgments from DPO and CM-PRM?
5 comments
Response to Reviewer z5Ca (Part2)
Note: This is a continuation of our Response to Weakness 1 / Question 1.
- Rationale for Trajectory-Wise Control
Regarding Eq. (11), completing the square does not undermine its design rationale; rather, it explicitly reveals the structure of trajectory-wise control. For margin-only objectives, jointly shifting and leaves the loss unchanged, so their common drift remains unconstrained. CM instead controls their absolute positions separately. Thus, the “controlled margin” does not merely specify a fixed gap between two targets; it separately regulates reinforcement and suppression scales through trajectory-wise feedback.
Response to Question 2
Regarding continued SFT, we added a second-stage control trained only on correct-verdict trajectories, denoted CM-PRM (SFT+SFT). Starting from the same SFT checkpoint and matching the second-stage training budget, this control achieves 59.3 on PRMBench and 62.7 on ProcessBench, compared with CM-PRM’s 69.1 and 71.3, yielding relative improvements of 16.5% and 13.7%. Under this setting, simply extending SFT on correct-verdict trajectories is insufficient to achieve comparable performance; using optimization signals from both correct and incorrect trajectories is more effective.
Regarding GRPO, we consider three aspects.
- A verifiable reward does not make GRPO a direct replacement for our optimization design. Online learning depends on the current policy sampling informative trajectories. When correct evaluations are difficult for the model to generate, obtaining useful positive signals requires additional exploration, whereas existing offline data can provide this supervision directly. Standard GRPO also relies on within-group relative rewards. When every trajectory in a group receives the same correctness reward, reward differences cannot provide a discriminative signal, whereas offline preference pairs directly supply contrasts between correct and incorrect verdicts.
- GRPO rollouts incur additional computational costs. GRPO requires repeated online generation of multiple analysis and verdict trajectories, followed by reward and group-advantage computation. Consequently, even with the same backbone and prompts, its total cost cannot be equated with offline training solely by matching parameter-update counts; generation costs must also be included.
- The practical budget for PRM training must remain bounded. Our goal is to use existing offline supervision to construct an evaluator for subsequent reasoning search and reinforcement learning at limited cost, rather than introducing an additional online RL stage during reward-model construction. Given our available resources, we have not conducted a GRPO comparison.
Our controlled comparisons examine the value of improving the objective under fixed offline data and training budgets; we do not claim that CM outperforms all online training methods.
Response to Weakness 2 / Question 3
| Threshold | CM-PRM | DPO |
|---|---|---|
| 0.5 | 0.713 | 0.668 |
| 0.3 | 0.582 | 0.524 |
| 0.7 | 0.638 | 0.602 |
Thank you for raising the possibility that a threshold shift could explain part of the improvement. We aim for a PRM that can be applied directly across tasks, rather than requiring threshold selection using validation labels from each new benchmark. Following R-PRM, all methods therefore use the same 0.5 majority-vote threshold, without separate tuning for ProcessBench.
To further examine threshold sensitivity, we added comparisons at thresholds of 0.3, 0.5, and 0.7. CM-PRM outperforms DPO at all three thresholds, with relative improvements of 11.1%, 6.7%, and 6.0%, respectively. Both methods also attain their highest scores among the tested thresholds at 0.5. Thus, CM-PRM’s advantage is not confined to the default threshold; it is reproduced across all three tested settings.
Response to Weakness 2 / Question 4
The 128K preference subset was randomly sampled from R-PRM’s 269K examples. We additionally trained DPO and CM-PRM on the complete 269K preference dataset, starting from the same SFT checkpoint. As shown above, CM-PRM-full achieves 61.8 on PRMBench versus DPO-full’s 61.1 and outperforms DPO-full on all four ProcessBench subsets.
Full-data training does not improve either method’s absolute scores. One possible explanation is distribution mismatch in offline preference optimization: stronger fitting to the offline preference data could exacerbate the mismatch between the current policy and the data-generating policy, potentially impairing generalization. These results do not support the view that adding data necessarily improves performance; meanwhile, CM retains its advantage over DPO under full-data training. We will include the sampling procedure and full-data results in the revision.
Response to Reviewer z5Ca (Part3)
Response to Weakness 2 / Question 5
We added a controlled comparison using the Qwen-Math-1.5B backbone. With the same SFT initialization, data, and training settings, CM improves PRMBench performance from DPO’s 55.3 to 56.2, a relative gain of 1.6%, and improves average ProcessBench performance from 47.3 to 49.4, a relative gain of 4.4%. These results provide additional evidence at a model scale beyond the original 7B backbone.
Response to Weakness 2 / Question 6
To test whether models overly rely on familiar local expressions, we conducted a meaning-preserving rewriting experiment on PRMBench’s deception subset. This subset tests whether PRMs can identify reasoning that appears plausible but contains subtle errors, requiring them to distinguish surface form from actual correctness. We selected all positively labeled examples, 7,881 in total, and used DeepSeek-v4 to replace common terms, formulas, and symbols in the problem statements with equivalent but less familiar expressions, preserving mathematical meaning and step correctness. We then compared verdict accuracy on the same examples before and after rewriting. Positive examples were selected to test whether unfamiliar expressions alone would cause a model to incorrectly reject a valid step.
| Method | Original Accuracy | Rewritten Accuracy |
|---|---|---|
| CM-PRM | 86.7 | 84.9 |
| DPO | 83.8 | 74.5 |
CM-PRM’s accuracy decreases from 86.7 to 84.9, a relative drop of 2.1%, whereas DPO’s decreases from 83.8 to 74.5, a relative drop of 11.1%. CM-PRM’s smaller degradation indicates that its judgments are less affected by changes in local surface forms. This provides systematic behavioral evidence beyond illustrative cases, supporting its greater robustness to meaning-preserving rewriting.
Response to Reviewer z5Ca (Part1)
We thank the reviewer for the careful review and for recognizing our analysis of the problem, empirical gains, and controlled ablations. We address your concerns point by point below.
Response to Weakness 1 / Question 1
We thank the reviewer for suggesting Cal-DPO, NCA, and DPOP as additional baselines. We further examined these works and their training mechanisms, and acknowledge that their objectives, although not originally developed for PRMs, are applicable to PRM preference learning. However, applicability does not imply identical optimization constraints or effectiveness. We therefore added objective-level analyses and direct comparisons under matched conditions, which show that CM’s objective configuration is more effective in our mathematical process-evaluation setting.
Let Δ=hw−hl and [z]+=max(z,0). The two losses are
=-\log\sigma(\Delta)+(h_w-a)^2+(h_l+a)^2, \qquad \ell_{\mathrm{DPOP}} =-\log\sigma\left(b[\Delta-\lambda[-h_w]_+]\right).$$ Cal-DPO retains the Bradley-Terry (BT) pairwise ranking term, whereas DPOP adds a one-sided penalty when the chosen likelihood falls below the reference likelihood. Both retain gradient-coefficient dependence on the relative margin, preserving the coupling identified in our analysis of Challenge 1: Synchronous Collapse. Moreover, DPOP’s additional penalty is inactive when the chosen likelihood reaches or exceeds the reference level, leaving no continuous individual control over both absolute scales. In contrast, each CM coefficient responds separately to its own log-ratio, so reinforcement of correct-verdict trajectories is no longer jointly determined by the relative gap between the two trajectories. - **CM-PRM vs. NCA** The pairwise-preference version of NCA uses $$\ell_{\mathrm{NCA}} =-\log\sigma(r_w)-\tfrac12\log\sigma(-r_w)-\tfrac12\log\sigma(-r_l), \qquad \frac{\partial\ell_{\mathrm{NCA}}}{\partial r_l} =\frac{1}{2}\sigma(r_l)>0,$$ where $r_i$ denotes a scaled log-ratio. Along the rejected coordinate, decreasing $r_l$ always reduces the loss. There is no finite negative target or corrective feedback that reverses excessive suppression, leaving incomplete two-sided scale control against Challenge 2: Log-ratio Inflation. CM instead sets finite targets on both sides and reverses the corresponding feedback after a target is crossed. This provides reinforcement signals for learning correct-verdict trajectories while limiting excessive suppression of incorrect-verdict trajectories, rather than primarily expanding the margin by lowering rejected likelihoods. - **Empirical Analysis** We added controlled comparisons with all three baselines. All methods share the same SFT checkpoint, reference model, exactly the same 128K preference examples, and training/validation splits. Except for the objective and its specific hyperparameters, the training budget, learning rate, batch size, LoRA configuration, and evaluation settings are identical. Objective-specific hyperparameters and checkpoints are selected using the same validation set and F1 criterion, without test-set tuning. CM-PRM achieves 69.1 on PRMBench, outperforming Cal-DPO, NCA, and DPOP by 13.1%, 12.9%, and 12.0%, respectively. On ProcessBench, it achieves 71.3, with corresponding relative improvements of 8.0%, 12.5%, and 14.1%. These results demonstrate that, under the same supervision and training budget, CM-PRM is more effective than these three objectives. Its joint control design provides practical benefits rather than being an equivalent substitution for an existing absolute log-ratio objective.