- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes Subset Champion Marginalization (SCM), a training objective for process reward models (PRMs) designed to improve solution selection. The central observation is that not all comparisons between correct and incorrect trajectories are equally important: their impact depends on which other candidates are available during inference.
SCM assigns each correct–incorrect trajectory pair a weight based on the probability that the two trajectories are the highest-scoring available candidates within their respective correctness classes. The paper derives an exact representation of conditional selection risk using these weights and constructs a differentiable pairwise ranking objective. It analytically aggregates comparison weights across multiple best-of-N budgets and combines the resulting trajectory-level loss with standard step-level PRM supervision, without changing the inference-time scoring rule.
The authors evaluate SCM using three reward-model backbones ranging from 0.6B to 7B parameters on four mathematical reasoning benchmarks, including MATH-500 and GSM-Plus, through best-of-N selection and beam search. SCM generally improves selection performance over existing and matched PRM baselines. On MATH-500, the 7B model improves macro area under the log-budget accuracy curve (AUBC) by 2.42 percentage points over the strongest matched baseline. Ablations also show improvements over uniform pairwise weighting, although the additional gain from SCM's proposed weighting is smaller.
Critical strengths
-
Principled, decision-relevant weighting of trajectory comparisons: The paper provides a useful formulation of PRM training in terms of downstream selection risk. Rather than treating all correct–incorrect trajectory pairs equally, SCM derives comparison weights from the probability that a pair determines the selected solution under a candidate-availability distribution. The resulting objective accounts for multiple best-of-N budgets without modifying inference. This is a meaningful contribution because it aligns training more closely with how PRMs are used for solution selection.
-
Consistent empirical improvements across evaluation settings: SCM generally outperforms matched PRM baselines across different model sizes, candidate generators, and mathematical reasoning benchmarks. The improvements also extend to PRM-guided beam search, suggesting that the learned scores are useful beyond reranking completed trajectories. The inclusion of uniform pairwise weighting and other ablations helps isolate the contribution of the proposed weighting scheme, although its incremental advantage is relatively modest.
Critical weaknesses
-
Statistical reliability of the proposed weighting scheme: While SCM improves performance over standard PRM baselines, much of the improvement may come from adding trajectory-level pairwise ranking supervision rather than the proposed decision-relevant weighting. On MATH-500, SCM achieves an AUBC of 49.26 compared with 48.31 for uniform pairwise weighting, an improvement of only 0.95 percentage points. The paper does not report multiple-seed results or confidence intervals to establish whether this difference is statistically reliable. This is consequential because the weighting scheme is the paper's primary methodological contribution.
-
Unclear impact on step-level PRM quality: SCM introduces a trajectory-level ranking objective that backpropagates through the PRM's step-level predictions. While this improves downstream solution selection, the paper does not evaluate whether it affects the model's ability to predict its original step-level supervision targets. This matters because PRMs are also used to evaluate intermediate reasoning steps, and improvements in trajectory ranking do not necessarily imply improvements in step-level prediction quality.
Questions for authors
-
Statistical significance of the proposed weighting scheme (experimental clarification): On MATH-500, SCM improves AUBC from 48.31 with uniform pairwise weighting to 49.26. Could the authors report variability across training seeds or confidence intervals for this comparison? Given that decision-relevant weighting is the primary methodological contribution, establishing whether its improvement over uniform weighting is reliable would materially affect my assessment.
-
Contribution of budget-aware weighting (experimental clarification): How much of SCM's improvement comes specifically from aggregating comparison weights across different candidate budgets? For example, how does training with weights derived from a single budget (such as N=32 or N=128) compare with the proposed multi-budget objective, both in overall AUBC and at individual evaluation budgets? This would help establish whether optimizing across candidate-set sizes is an important component of the observed gains.
-
Effect on step-level prediction quality (experimental clarification): Since the trajectory-ranking loss backpropagates through the PRM's step-level scores, does adding SCM affect prediction quality on the original rollout-derived step labels compared with step-only training? A comparison using step-level predictive metrics would clarify whether improved trajectory selection comes at the expense of the original step-level supervision objective.
6 comments
Response to Reviewer 7QBr (Part II)
Question 2: How much of SCM's improvement comes specifically from aggregating comparison weights across different candidate budgets?
A matched single-budget comparison directly tests the role of budget aggregation. We conducted two additional training runs using only or only under the same with-replacement (WR) law, changing only the training-budget distribution. Both were evaluated across the complete BoN curve.
Table 2 shows that the default mixture reaches 49.26% macro AUBC, exceeding single-budget training by 1.35 and 1.65 pp, respectively. It also leads at every evaluation budget, supporting the benefit of combining candidate-set sizes. We have added the full curves to Table 18 in Appendix B.7 of the revised manuscript, with the training protocol in Appendix C.6.
Table 2. MATH-500 BoN accuracy and AUBC (%); equal-generator means over three pools.
| Training | Bo8 | Bo16 | Bo32 | Bo64 | Bo128 | AUBC |
|---|---|---|---|---|---|---|
| Only | 42.73 | 46.00 | 48.80 | 50.20 | 50.53 | 47.91 |
| Only | 42.67 | 45.67 | 48.40 | 49.67 | 50.73 | 47.61 |
| SCM (multi-budget) | 44.67 | 47.13 | 49.67 | 51.67 | 52.47 | 49.26 |
Response to Reviewer 7QBr (Part III)
Weakness 2: While this improves downstream solution selection, the paper does not evaluate whether it affects the model's ability to predict its original step-level supervision targets.
Question 3: Since the trajectory-ranking loss backpropagates through the PRM's step-level scores, does adding SCM affect prediction quality on the original rollout-derived step labels compared with step-only training?
Thank you for raising this complementary perspective on PRM quality. Better trajectory selection should also be examined alongside prediction of the original step labels. Our objective in Section 3.5 retains step-level BCE alongside the trajectory loss. Following your suggestion, we additionally evaluated SCM and Step-only on the original P0 TEST partition: 108,119 valid steps from 19,176 trajectories and 691 question groups. These samples were unused for training or checkpoint selection; both models use their final step-1554 checkpoints.
SCM shows small improvements in all three metrics in Table 3. BCE and Brier assess probability-prediction quality through strictly proper scoring rules [1], while AUROC measures discrimination. Thus, the observed gain in trajectory selection is accompanied by preserved and slightly improved prediction of the original step targets. We have added these new results to Table 22 in Appendix B.8 of the revised manuscript and described the held-out protocol in Appendix C.7.
Table 3. Prediction of held-out original rollout-derived step labels.
| Method | BCE ↓ | AUROC ↑ | Brier ↓ |
|---|---|---|---|
| Step-only | 0.284809 | 0.939929 | 0.087530 |
| SCM | 0.282007 | 0.940426 | 0.086462 |
To complement the original-label evaluation, we also assessed independent step-level verification on ProcessBench using final checkpoints and official step boundaries. Table 4 shows the highest macro F for SCM, 38.92, compared with 36.28 for CRM and 31.36 for PQM. The full results are reported in Table 21 in Appendix B.8, with the protocol in Appendix C.7.
Table 4. ProcessBench F (%): the harmonic mean of fully-correct recognition accuracy and first-error localization accuracy. Macro averages the four subsets.
| Method | GSM8K | MATH | OlympiadBench | OmniMath | Macro F |
|---|---|---|---|---|---|
| PQM | 69.94 | 33.32 | 6.90 | 15.26 | 31.36 |
| CRM | 61.85 | 44.48 | 20.64 | 18.14 | 36.28 |
| SCM | 63.27 | 45.99 | 24.42 | 22.01 | 38.92 |
References
[1] Tilmann Gneiting and Adrian E. Raftery. Strictly proper scoring rules, prediction, and estimation. Journal of the American Statistical Association, 102(477):359–378, 2007. doi: 10.1198/016214506000001437.
Response to Reviewer 7QBr (Part I)
We appreciate your careful reading and constructive questions. We are deeply encouraged the reviewer recognized the strengths of our work from various perspectives. Below we respond to your comments and questions in detail.
We agree that the weighting gain should be examined across training seeds. Adding trajectory supervision and choosing its comparison weights are separate changes, so this distinction is central to assessing SCM. In the original manuscript, Section 4.4 separates the two effects using matched training objectives. Adding uniform pairwise supervision raises macro AUBC from 46.98% for Step-only to 48.31%; replacing uniform weights with SCM raises it further to 49.26%. The latter 0.95 pp gain isolates the weighting contribution while retaining the same ranking loss and step supervision.
To test whether this gain persists across training seeds, we performed additional runs with seeds 2026 and 2027 alongside the original seed 42, keeping all other settings fixed. SCM improves macro AUBC in all three seeds, with a mean paired gain of 0.81 ± 0.19 pp (mean ± sample SD). Thus, the weighting gain is consistent across these runs, beyond the improvement from adding trajectory supervision. Table 1 reports the complete BoN curves. We have summarized this new experiment in Section 4.4 and added the full results to Table 16 in Appendix B.6 of the revised manuscript. The additions are marked in blue.
Table 1. Best-of-N accuracy (%) on MATH-500, reported as mean ± sample SD. All methods use Qwen2.5-Math-7B-Instruct. Bold/underline mark the highest/second-highest distinct means within each sampling policy.