acceptodds
Under review as a conference paper at ICLR 2027

SCM: DECISION-RELEVANT COMPARISONS FOR PROCESS REWARD MODELING

14 views

Abstract

Process reward models (PRMs) guide reasoning by scoring intermediate steps and selecting among candidate solutions. Yet uniform pair weights ignore how often comparisons decide selection, while successful full-pool selection can conceal errors that become decisive when a stronger correct solution is absent. To train for these decisions, we study selection risk conditional on both correct and incorrect solutions being available. We propose Subset Champion Marginalization (SCM), an objective that weights comparisons by the joint distribution of the best available correct and incorrect solutions. With within-class orders fixed, these weights uniquely represent selection risk as linear comparison costs. SCM marginalizes these costs analytically across candidate budgets and combines the resulting trajectory loss with step supervision, using existing labels and the same inference-time scoring rule. We evaluate three reward-model backbones from 0.6B to 7B on four mathematical reasoning benchmarks through best-of- selection and beam search. At 7B, SCM improves macro area under the log-budget accuracy curve on MATH-500 by 2.42 percentage points over the best matched baseline and 1.60 points over the best released PRM evaluated.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Process Reward Modeling, Large Language Model Reasoning, Test-Time Scaling, Best-of-N Selection

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_7QBrOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes Subset Champion Marginalization (SCM), a training objective for process reward models (PRMs) designed to improve solution selection. The central observation is that not all comparisons between correct and incorrect trajectories are equally important: their impact depends on which other candidates are available during inference.

SCM assigns each correct–incorrect trajectory pair a weight based on the probability that the two trajectories are the highest-scoring available candidates within their respective correctness classes. The paper derives an exact representation of conditional selection risk using these weights and constructs a differentiable pairwise ranking objective. It analytically aggregates comparison weights across multiple best-of-N budgets and combines the resulting trajectory-level loss with standard step-level PRM supervision, without changing the inference-time scoring rule.

The authors evaluate SCM using three reward-model backbones ranging from 0.6B to 7B parameters on four mathematical reasoning benchmarks, including MATH-500 and GSM-Plus, through best-of-N selection and beam search. SCM generally improves selection performance over existing and matched PRM baselines. On MATH-500, the 7B model improves macro area under the log-budget accuracy curve (AUBC) by 2.42 percentage points over the strongest matched baseline. Ablations also show improvements over uniform pairwise weighting, although the additional gain from SCM's proposed weighting is smaller.

Critical strengths
  1. Principled, decision-relevant weighting of trajectory comparisons: The paper provides a useful formulation of PRM training in terms of downstream selection risk. Rather than treating all correct–incorrect trajectory pairs equally, SCM derives comparison weights from the probability that a pair determines the selected solution under a candidate-availability distribution. The resulting objective accounts for multiple best-of-N budgets without modifying inference. This is a meaningful contribution because it aligns training more closely with how PRMs are used for solution selection.

  2. Consistent empirical improvements across evaluation settings: SCM generally outperforms matched PRM baselines across different model sizes, candidate generators, and mathematical reasoning benchmarks. The improvements also extend to PRM-guided beam search, suggesting that the learned scores are useful beyond reranking completed trajectories. The inclusion of uniform pairwise weighting and other ablations helps isolate the contribution of the proposed weighting scheme, although its incremental advantage is relatively modest.

Critical weaknesses
  1. Statistical reliability of the proposed weighting scheme: While SCM improves performance over standard PRM baselines, much of the improvement may come from adding trajectory-level pairwise ranking supervision rather than the proposed decision-relevant weighting. On MATH-500, SCM achieves an AUBC of 49.26 compared with 48.31 for uniform pairwise weighting, an improvement of only 0.95 percentage points. The paper does not report multiple-seed results or confidence intervals to establish whether this difference is statistically reliable. This is consequential because the weighting scheme is the paper's primary methodological contribution.

  2. Unclear impact on step-level PRM quality: SCM introduces a trajectory-level ranking objective that backpropagates through the PRM's step-level predictions. While this improves downstream solution selection, the paper does not evaluate whether it affects the model's ability to predict its original step-level supervision targets. This matters because PRMs are also used to evaluate intermediate reasoning steps, and improvements in trajectory ranking do not necessarily imply improvements in step-level prediction quality.

Questions for authors
  1. Statistical significance of the proposed weighting scheme (experimental clarification): On MATH-500, SCM improves AUBC from 48.31 with uniform pairwise weighting to 49.26. Could the authors report variability across training seeds or confidence intervals for this comparison? Given that decision-relevant weighting is the primary methodological contribution, establishing whether its improvement over uniform weighting is reliable would materially affect my assessment.

  2. Contribution of budget-aware weighting (experimental clarification): How much of SCM's improvement comes specifically from aggregating comparison weights across different candidate budgets? For example, how does training with weights derived from a single budget (such as N=32 or N=128) compare with the proposed multi-budget objective, both in overall AUBC and at individual evaluation budgets? This would help establish whether optimizing across candidate-set sizes is an important component of the observed gains.

  3. Effect on step-level prediction quality (experimental clarification): Since the trajectory-ranking loss backpropagates through the PRM's step-level scores, does adding SCM affect prediction quality on the original rollout-derived step labels compared with step-only training? A comparison using step-level predictive metrics would clarify whether improved trajectory selection comes at the expense of the original step-level supervision objective.

6 comments
AuthorsOpenReview ↗

Response to Reviewer 7QBr (Part I)

We appreciate your careful reading and constructive questions. We are deeply encouraged the reviewer recognized the strengths of our work from various perspectives. Below we respond to your comments and questions in detail.

Weakness 1: While SCM improves performance over standard PRM baselines, much of the improvement may come from adding trajectory-level pairwise ranking supervision rather than the proposed decision-relevant weighting.

Question 1: Could the authors report variability across training seeds or confidence intervals for this comparison?

We agree that the weighting gain should be examined across training seeds. Adding trajectory supervision and choosing its comparison weights are separate changes, so this distinction is central to assessing SCM. In the original manuscript, Section 4.4 separates the two effects using matched training objectives. Adding uniform pairwise supervision raises macro AUBC from 46.98% for Step-only to 48.31%; replacing uniform weights with SCM raises it further to 49.26%. The latter 0.95 pp gain isolates the weighting contribution while retaining the same ranking loss and step supervision.

To test whether this gain persists across training seeds, we performed additional runs with seeds 2026 and 2027 alongside the original seed 42, keeping all other settings fixed. SCM improves macro AUBC in all three seeds, with a mean paired gain of 0.81 ± 0.19 pp (mean ± sample SD). Thus, the weighting gain is consistent across these runs, beyond the improvement from adding trajectory supervision. Table 1 reports the complete BoN curves. We have summarized this new experiment in Section 4.4 and added the full results to Table 16 in Appendix B.6 of the revised manuscript. The additions are marked in blue.

Table 1. Best-of-NN accuracy (%) on MATH-500, reported as mean ± sample SD. All methods use Qwen2.5-Math-7B-Instruct. Bold/underline mark the highest/second-highest distinct means within each sampling policy.

Sampling policyMethod@8@16@32@64@128
Llama3-70B-InstructPQM51.33±0.12‾\underline{51.33 \pm 0.12}52.67±0.50‾\underline{52.67 \pm 0.50}52.67 ± 1.0153.67 ± 0.3154.40 ± 0.20
CRM49.80 ± 1.6451.60 ± 1.5652.33 ± 0.9951.60 ± 1.7450.93 ± 2.00
Uniform pairs51.80 ± 1.5652.67±1.15‾\underline{52.67 \pm 1.15}53.60±1.44‾\underline{53.60 \pm 1.44}54.67 ± 1.6255.00±0.92‾\underline{55.00 \pm 0.92}
SCM51.80 ± 0.8752.87 ± 0.9053.87 ± 0.9054.60±1.11‾\underline{54.60 \pm 1.11}55.67 ± 1.03
MetaMath-MistralPQM42.40 ± 1.2245.13 ± 0.1248.53 ± 0.5051.33 ± 0.6151.60 ± 0.72
CRM42.80±0.92‾\underline{42.80 \pm 0.92}46.93 ± 1.1749.47 ± 1.1051.40 ± 1.9151.80 ± 1.22
Uniform pairs42.73 ± 0.1247.47±0.64‾\underline{47.47 \pm 0.64}50.47±1.03‾\underline{50.47 \pm 1.03}53.00±0.20‾\underline{53.00 \pm 0.20}53.87±1.01‾\underline{53.87 \pm 1.01}
SCM43.87 ± 0.6448.60 ± 1.0651.80 ± 1.0054.53 ± 0.6454.93 ± 1.40
Muggle-Math-13BPQM34.60 ± 0.3539.27±0.12‾\underline{39.27 \pm 0.12}40.80 ± 1.4443.13±0.83‾\underline{43.13 \pm 0.83}43.53 ± 0.83
CRM36.13±0.61‾\underline{36.13 \pm 0.61}39.20 ± 0.6939.80 ± 0.2042.73 ± 0.6442.67 ± 0.81
Uniform pairs35.40 ± 0.4039.27±1.03‾\underline{39.27 \pm 1.03}41.13±0.61‾\underline{41.13 \pm 0.61}43.13±0.64‾\underline{43.13 \pm 0.64}44.13±0.70‾\underline{44.13 \pm 0.70}
SCM36.33 ± 0.4639.67 ± 0.3142.20 ± 0.5344.53 ± 0.1245.27 ± 0.61
AuthorsOpenReview ↗

Response to Reviewer 7QBr (Part II)

Question 2: How much of SCM's improvement comes specifically from aggregating comparison weights across different candidate budgets?

A matched single-budget comparison directly tests the role of budget aggregation. We conducted two additional training runs using only N=32N=32 or only N=128N=128 under the same with-replacement (WR) law, changing only the training-budget distribution. Both were evaluated across the complete BoN curve.

Table 2 shows that the default mixture reaches 49.26% macro AUBC, exceeding single-budget training by 1.35 and 1.65 pp, respectively. It also leads at every evaluation budget, supporting the benefit of combining candidate-set sizes. We have added the full curves to Table 18 in Appendix B.7 of the revised manuscript, with the training protocol in Appendix C.6.

Table 2. MATH-500 BoN accuracy and AUBC (%); equal-generator means over three pools.

TrainingBo8Bo16Bo32Bo64Bo128AUBC
Only N=32N=3242.7346.0048.8050.2050.5347.91
Only N=128N=12842.6745.6748.4049.6750.7347.61
SCM (multi-budget)44.6747.1349.6751.6752.4749.26
AuthorsOpenReview ↗

Response to Reviewer 7QBr (Part III)

Weakness 2: While this improves downstream solution selection, the paper does not evaluate whether it affects the model's ability to predict its original step-level supervision targets.

Question 3: Since the trajectory-ranking loss backpropagates through the PRM's step-level scores, does adding SCM affect prediction quality on the original rollout-derived step labels compared with step-only training?

Thank you for raising this complementary perspective on PRM quality. Better trajectory selection should also be examined alongside prediction of the original step labels. Our objective in Section 3.5 retains step-level BCE alongside the trajectory loss. Following your suggestion, we additionally evaluated SCM and Step-only on the original P0 TEST partition: 108,119 valid steps from 19,176 trajectories and 691 question groups. These samples were unused for training or checkpoint selection; both models use their final step-1554 checkpoints.

SCM shows small improvements in all three metrics in Table 3. BCE and Brier assess probability-prediction quality through strictly proper scoring rules [1], while AUROC measures discrimination. Thus, the observed gain in trajectory selection is accompanied by preserved and slightly improved prediction of the original step targets. We have added these new results to Table 22 in Appendix B.8 of the revised manuscript and described the held-out protocol in Appendix C.7.

Table 3. Prediction of held-out original rollout-derived step labels.

MethodBCE ↓AUROC ↑Brier ↓
Step-only0.2848090.9399290.087530
SCM0.2820070.9404260.086462

To complement the original-label evaluation, we also assessed independent step-level verification on ProcessBench using final checkpoints and official step boundaries. Table 4 shows the highest macro F for SCM, 38.92, compared with 36.28 for CRM and 31.36 for PQM. The full results are reported in Table 21 in Appendix B.8, with the protocol in Appendix C.7.

Table 4. ProcessBench F (%): the harmonic mean of fully-correct recognition accuracy and first-error localization accuracy. Macro averages the four subsets.

MethodGSM8KMATHOlympiadBenchOmniMathMacro F
PQM69.9433.326.9015.2631.36
CRM61.8544.4820.6418.1436.28
SCM63.2745.9924.4222.0138.92

References

[1] Tilmann Gneiting and Adrian E. Raftery. Strictly proper scoring rules, prediction, and estimation. Journal of the American Statistical Association, 102(477):359–378, 2007. doi: 10.1198/016214506000001437.

Reviewer_cuSgOpenReview ↗
Rating
2
Confidence
3
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes Subset Champion Marginalization (SCM), a training objective for process reward models (PRMs) that aims to better align trajectory-level supervision with downstream candidate selection. The key observation is that not all correct–incorrect trajectory comparisons are equally important: their relevance depends on how likely they are to determine the final selection under different candidate subsets. SCM addresses this by deriving comparison weights from the joint distribution of the highest-scoring correct and incorrect candidates. The authors provide theoretical results characterizing these weights, establishing their relationship to conditional selection risk, and deriving an analytical marginalization across candidate budgets. The resulting trajectory-level objective is combined with conventional step-level supervision without changing the inference-time scoring rule.

The method is evaluated using PRMs ranging from 0.6B to 7B parameters on several mathematical reasoning benchmarks, including MATH-500, GSM-Plus, Gaokao2023, and OlympiadBench. Experiments cover both best-of-N selection and PRM-guided beam search. SCM generally outperforms matched PQM and CRM baselines and achieves competitive results against several released PRMs. Ablation studies further examine the contributions of step supervision, trajectory supervision, and different comparison-weighting strategies.

Critical strengths

The paper identifies an important limitation of conventional pairwise supervision for PRMs: comparisons that appear unimportant in the full candidate pool may become decisive when stronger candidates are unavailable. SCM addresses this issue by deriving comparison weights from the joint distribution of the best available correct and incorrect trajectories. The theoretical characterization of these weights, their connection to conditional selection risk, and the analytical marginalization across candidate budgets provide a principled foundation for the proposed objective. I consider this the paper's strongest contribution, as it goes beyond introducing another heuristic ranking loss and offers a clear explanation of why particular comparisons should receive greater emphasis.

The authors evaluate SCM across multiple PRM backbones and mathematical reasoning benchmarks, covering both best-of-N selection and beam search. Importantly, SCM consistently improves over matched PQM and CRM baselines at 7B and 1.7B, with a 2.42 percentage-point improvement in MATH-500 macro AUBC over CRM at 7B. The ablations also help isolate the contributions of trajectory supervision and comparison weighting. These results provide credible evidence that the proposed objective is practically useful, even though its advantages over stronger released PRMs are less consistent.

Critical weaknesses
  1. The novelty relative to existing ranking objectives is not sufficiently established. SCM weights trajectory comparisons based on their likelihood of affecting the final selection. While deriving these weights from the champion distribution is interesting, the general idea of optimizing ranking-related decisions has been explored in methods such as LambdaRank/LambdaLoss and ListMLE. Although these methods are discussed, they are not included in the experimental comparisons. It remains unclear whether SCM offers meaningful advantages over existing ranking-aware objectives.

  2. The theoretical assumptions may not fully reflect practical search settings. The analysis assumes a predefined, score-independent distribution over candidate subsets. In practice, candidate generation is often nonuniform and adaptive, particularly in beam search, where PRM scores directly influence which trajectories are retained. While the experiments consider different sampling protocols on fixed candidate pools, it is unclear how well these results extend to the distribution shifts induced by actual search.

  3. Several design choices lack sufficient justification and ablation. The budget mixture in SCM is chosen to align with the log-budget AUBC metric, raising questions about its robustness to other inference-budget distributions. The effects of the softplus temperature, trajectory-loss weight, and its scheduling are also not thoroughly examined. Given that SCM combines two supervision objectives, a more systematic analysis of these choices would help clarify their contributions.

  4. The evaluation is limited in scope and lacks a clear efficiency comparison. Despite covering multiple benchmarks and model sizes, the experiments focus exclusively on mathematical reasoning. It remains unclear whether SCM generalizes to tasks with noisier correctness signals or different candidate distributions, such as code generation. In addition, although analytical marginalization avoids explicit subset sampling, it introduces additional pairwise computations. A direct comparison of training costs would help assess the practical trade-offs.

Overall, I find the work technically interesting and potentially useful, but I am not yet convinced that its methodological novelty and practical generality have been sufficiently demonstrated. I therefore lean toward a weak reject, although stronger evidence addressing these concerns could change my assessment.

Questions for authors
  1. Could the authors clarify what SCM provides beyond existing ranking-aware training objectives? In particular, how would SCM compare with a standard pairwise ranking loss or a representative listwise objective, such as ListMLE, when combined with the same step-level supervision? This would help isolate the contribution of the champion-based weights.

  2. How sensitive is SCM to the assumed candidate distribution? The theoretical derivation uses a score-independent sampling law, but the candidate distribution in beam search depends on the PRM itself. Have the authors considered testing the method under different generation temperatures, nonuniform sampling probabilities, or more substantial changes in the candidate distribution?

  3. Does SCM generalize beyond mathematical reasoning, and what is its practical training overhead? An experiment on another domain, such as code generation, would help assess whether the approach depends on the characteristics of mathematical reasoning datasets. It would also be helpful to report end-to-end training throughput or GPU-hours compared with simpler pairwise objectives.

4 comments
AuthorsOpenReview ↗

Response to Reviewer cuSg (Part I)

We appreciate your careful reading and constructive feedback.We are deeply encouraged the reviewer recognized the strengths of our work from various perspectives. Below we respond to your comments and questions in detail.

Weakness 1: The novelty relative to existing ranking objectives is not sufficiently established.

Question 1: Could the authors clarify what SCM provides beyond existing ranking-aware training objectives?

Thank you for this suggestion. ListMLE fits target permutations [1]; LambdaRank uses metric-weighted pairwise updates [2, 3]. SCM derives comparison costs from selection risk, following decision-focused learning.

Sections 3.1–3.2 prove uniqueness of these linear costs for fixed within-class orders. For a fixed law, let μij\mu_{ij} be the joint probability that correct ii and incorrect jj are class champions, conditional on both classes being available. Then

Rμ(S)=∑i,jμij1j outranks i.R_\mu(S)=\sum_{i,j}\mu_{ij}\mathbf{1}\\{j\text{ outranks }i\\}.

An adjacent cross-class swap changes risk by μij\mu_{ij} in magnitude, identifying its cost. Section 3.5 exactly averages the surrogate and its gradient within strict within-class ordering regions, without subset sampling.

Table 1 shows SCM outperforming random-tie ListMLE and full-list NDCG LambdaRank in all 15 matched settings. Uniform pairs isolates weighting. We added these results and definitions in Section 4.4, Table 6 and Appendix C.6, marked in blue.

Table 1. Best-of-NN accuracy (%) for matched ranking objectives on MATH-500.

Sampling policyMethod@8@16@32@64@128
Llama3-70B-InstructListMLE52.0052.6053.4053.2053.40
LambdaRank52.0053.2053.4054.6054.20
SCM52.8053.8054.8055.6056.80
MetaMath-MistralListMLE42.4047.2049.6051.8053.00
LambdaRank43.4047.2050.2052.6053.20
SCM44.6048.2051.8054.8054.80
Muggle-Math-13BListMLE33.8036.8038.8040.6041.00
LambdaRank36.0039.2041.8043.2044.40
SCM36.6039.4042.4044.6045.80

Weakness 3: Several design choices lack sufficient justification and ablation.

We agree that robustness to these design choices is important. Each run changes one factor. Tables 2–4 test the budget prior, off-grid budgets, temperature, loss weight, and schedule. All training variants outperform PQM/CRM in macro AUBC; off-grid gains are 1.33–2.07/1.20–2.80 pp. Results are in revised Section 4.4, Table 7 and Appendix B.7, with protocols in C.6.

Joint supervision achieves 49.26% versus 46.98%/47.75% for Step-only/Set-only. Single-budget N=32/128N=32/128 runs reach 47.91%/47.61%, below the default mixture. Defaults are τ=0.05\tau=0.05, λmax⁡=1\lambda_{\max}=1, and a 10%–20% ramp after step-only warm-up; Set-only ramps from training start.

Table 2. Sensitivity to the training budget prior on MATH-500.

PriorMixture weightsMacro AUBC (%)
Default(1,2,2,2,1)/8(1,2,2,2,1)/849.26
Uniform(1,1,1,1,1)/5(1,1,1,1,1)/548.57
Low-budget emphasis(16,8,4,2,1)/31(16,8,4,2,1)/3148.32
High-budget emphasis(1,2,4,8,16)/31(1,2,4,8,16)/3147.73

Table 3. MATH-500 BoN accuracy (%) at off-grid budgets. Macro equally averages generators.

Sampling policyMethod@10@20@40@80
LlamaPQM51.6052.8053.2054.20
CRM51.0052.4053.2052.20
SCM52.4053.8054.6055.60
MetaMathPQM43.0046.6051.0052.40
CRM42.4047.2050.0051.20
SCM46.2048.6052.2055.00
MugglePQM34.8040.0042.2043.00
CRM36.6040.2043.4044.00
SCM35.2041.0044.0045.20
MacroPQM43.1346.4748.8049.87
CRM43.3346.6048.8749.13
SCM44.6047.8050.2751.93

Table 4. Loss sensitivity on MATH-500; macro AUBC (%). One factor varies per block. Defaults bold.

(a) Temperature τ\tauAUBC  (b) Loss weight λmax⁡\lambda_{\max}AUBC  (c) Ramp intervalAUBC
0.02548.630.548.220%–20%48.35
0.0549.261.049.2610%–20%49.26
0.1048.612.047.71

References

[1] Fen Xia, Tie-Yan Liu, Jue Wang, Wensheng Zhang, and Hang Li. Listwise approach to learning to rank: Theory and algorithm. In Proceedings of the 25th International Conference on Machine Learning, pp. 1192–1199, 2008. doi: 10.1145/1390156.1390306.

[2] Christopher J. C. Burges, Robert Ragno, and Quoc Viet Le. Learning to rank with nonsmooth cost functions. In Advances in Neural Information Processing Systems, volume 19, pp. 193–200. MIT Press, 2006. doi: 10.7551/mitpress/7503.003.0029.

[3] Xuanhui Wang, Cheng Li, Nadav Golbandi, Michael Bendersky, and Marc Najork. The LambdaLoss framework for ranking metric optimization. In Proceedings of the 27th ACM International Conference on Information and Knowledge Management, pp. 1313–1322, 2018. doi: 10.1145/3269206.3271784.

AuthorsOpenReview ↗

Response to Reviewer cuSg (Part II)

Weakness 2: In practice, candidate generation is often nonuniform and adaptive, particularly in beam search, where PRM scores directly influence which trajectories are retained.

Question 2: How sensitive is SCM to the assumed candidate distribution?

We agree that the link between the sampling model and practical search needs to be made explicit. Nonuniformity differs from score dependence. Sections 3.1–3.3 allow any specified distribution over candidate collections, held fixed as scores change. For every mixed collection, only its best correct and best incorrect candidates determine selection; averaging this identity yields the champion-weighted selection risk. Uniform WR is a tractable instance, not a requirement. Appendix A.4 permits unequal probabilities; a fixed prefix is a degenerate collection law.

For the same labeled pool and within-class orders, let μ,ν\mu,\nu be the champion-pair distributions induced by two collection laws, each conditioned on mixedness. Section 3.3 proves

∣Rν(S)−Rμ(S)∣≤dsel(μ,ν)≤TV⁡(μ,ν).|R_\nu(S)-R_\mu(S)|\le d_{\mathrm{sel}}(\mu,\nu)\le\operatorname{TV}(\mu,\nu).

Here dseld_{\mathrm{sel}} maximizes the risk difference over realizable ranking errors, an integral probability metric [5]. Allowing all events gives TV⁡=12∑ij∣μij−νij∣\operatorname{TV}=\tfrac12\sum_{ij}|\mu_{ij}-\nu_{ij}|. The bound controls conditional-risk shifts through changes in decisive-comparison probabilities. Original Appendix B.5, Table 13 contains the following fixed-model comparison; SCM leads under every law.

Original Table 13. Macro AUBC (%).

LawPQMCRMSCM
Prefix46.7646.8449.26
WOR46.1546.1748.71
WR45.7845.8248.29

Question 2: Have the authors considered testing the method under different generation temperatures, nonuniform sampling probabilities, or more substantial changes in the candidate distribution?

We also appreciate your suggestion to test this explanation on newly generated candidates. We additionally generated 256 candidates per question with Qwen3-4B on all 60 AIME 2024/2025 questions at three temperatures. Verifiers share each pool and order; only temperature changes. Table 5 shows SCM matching or exceeding PQM in all 12 settings and CRM in 11 of 12. Results are in revised Appendix B.5, Table 15.

Table 5. Frozen reward models on AIME under different generation temperatures; accuracy (%).

TemperatureMethodBo16Bo64Bo128Bo256
0.4PQM18.3315.0013.3313.33
CRM21.6718.3316.6718.33
SCM25.0020.0018.3318.33
0.7PQM20.0016.6713.3315.00
CRM23.3316.6721.6718.33
SCM20.0021.6721.6718.33
1.0PQM18.3318.3318.3318.33
CRM20.0020.0018.3318.33
SCM20.0021.6720.0020.00

In beam search, the PRM ranks partial solutions and prunes beams, so earlier decisions change later candidates. This feedback is the distribution effect studied in sequential prediction [6]. With a frozen scorer, the champion identity still describes final selection from any realized mixed terminal collection. However, changing the scorer also changes the collection distribution: the fixed-law gradient does not capture this additional dependence, and single-winner analysis does not characterize top-k pruning.

The original Section 4.3, Table 3 tests SCM inside this adaptive procedure: its scores guide expansion and pruning, rather than merely reranking a shared terminal pool. With the same checkpoints as BoN, SCM improves over the stronger matched PQM/CRM baseline by 1.05–3.00 pp across all eight 7B-verifier configurations. These results demonstrate benefits when candidate availability is shaped by the scorer itself; the fixed-law theory explains the comparison objective, while beam search directly tests its effectiveness under that feedback.

References

[5] Alfred Müller. Integral probability metrics and their generating classes of functions. Advances in Applied Probability, 29(2):429–443, 1997. doi: 10.2307/1428011.

[6] Stéphane Ross, Geoffrey Gordon, and Drew Bagnell. A reduction of imitation learning and structured prediction to no-regret online learning. In Proceedings of the Fourteenth International Conference on Artificial Intelligence and Statistics, volume 15 of Proceedings of Machine Learning Research, pp. 627–635. PMLR, 2011.

AuthorsOpenReview ↗

Response to Reviewer cuSg (Part III)

Weakness 4: It remains unclear whether SCM generalizes to tasks with noisier correctness signals or different candidate distributions, such as code generation.

Question 3: Does SCM generalize beyond mathematical reasoning, and what is its practical training overhead?

Thank you for suggesting this broader evaluation. Our initial experiments focused on mathematical reasoning, where SCM improves BoN selection and beam search. Following your suggestion, we extended the study to test whether the same training principle remains effective in other reasoning domains.

We additionally trained each method jointly across physics, biology, business, and computer science using VersaPRM’s automatically annotated MMLU-Pro-CoT data. All use Qwen3-4B, rank-128 LoRA, three epochs, and final checkpoints; PQM/CRM retain their original losses and scoring rules. Evaluation uses exact uniform-WOR expectations on 570 held-out questions with at least 64 candidates each.

SCM has the highest overall accuracy at all three budgets, reaching 71.47% at Bo64, ahead of PQM/CRM by 1.69/6.52 pp. As Table 6 shows, it exceeds CRM in all 12 domain–budget settings and PQM in 9 of 12. These new results are now in revised Section 4.5, Table 8, with the protocol in Appendix C.7.

Table 6. Best-of-NN accuracy (%) across four domains: exact uniform-WOR expectations. Overall is weighted by question count.

DomainMethod@16@32@64
PhysicsPQM65.4767.3769.61
CRM62.7661.4059.54
SCM65.8568.3070.37
BiologyPQM81.7182.8383.08
CRM79.5580.5680.75
SCM80.5481.1380.96
BusinessPQM65.9867.1767.57
CRM67.1067.1565.76
SCM68.4370.6871.16
Computer sciencePQM59.5660.0260.48
CRM56.5956.2555.66
SCM63.0063.9164.56
OverallPQM67.7668.9269.78
CRM66.0865.8964.95
SCM69.1170.6871.47

Weakness 4: In addition, although analytical marginalization avoids explicit subset sampling, it introduces additional pairwise computations.

Question 3: It would also be helpful to report end-to-end training throughput or GPU-hours compared with simpler pairwise objectives.

Thank you for raising this practical question. Additional comparison costs should be assessed alongside the gains. We added complete-update profiling on four RTX 4090 GPUs with identical real batches and fully enabled trajectory losses. Timing covers forward, loss construction, backward, synchronization, and optimization.

SCM gains 0.95 pp in AUBC over Uniform pairs for 1.47% extra update time under the same two-pass implementation. It improves over ListMLE/LambdaRank at comparable costs. Against one-pass PQM/CRM, it gains 2.50/2.42 pp with 29.2% more update time; Table 7 also shows equal peak memory at the reported precision.

A shorter SCM run using 1,244 updates with the original data and schedule reaches 48.57% macro AUBC, above PQM, CRM, ListMLE, and LambdaRank. Based on the measured per-update times, this corresponds to approximately 1.03 times the update-time cost of full-length PQM/CRM training.

Additional scaling measurements fix eight questions per update. Increasing KK from 8 to 128 raises input volume 16×16\times and measured time 14.14×14.14\times; memory stays at 16.74–16.87 GiB/GPU and time within 0.2% of Uniform pairs. For a fixed budget grid, sorting and weight construction cost O(Klog⁡K+ab)O(K\log K+ab) beyond scoring, where a,ba,b count correct/incorrect candidates. The new timing and scaling results are in revised Appendix B.10, Tables 23–25.

Table 7. Complete-update training cost on four RTX 4090 GPUs.

MethodSeconds/update, mean ±\pm SDTokens/sPeak GiB/GPU
SCM10.771±0.08010.771\pm0.0805,97416.86
Uniform pairs10.615±0.01910.615\pm0.0196,06216.86
PQM8.335±0.0318.335\pm0.0317,71916.86
CRM8.336±0.0218.336\pm0.0217,71916.86
LambdaRank10.795±0.02610.795\pm0.0265,96016.86
ListMLE10.789±0.02810.789\pm0.0285,96416.86

Throughput counts non-padding tokens across all four GPUs; SD is across three round means, each based on eight timed updates after warm-up.

Dear Reviewers cuSg:

We wanted to touch base with you as the deadline for the author-reviewer discussion phase is approaching on November 18. We trust you've had the opportunity to review our rebuttal, and we would be more than happy to address any further concerns you have.

These days, we have also updated our draft to incorporate your suggestions. Your feedback has been instrumental in improving our work - thank you again!

Best,

Authors

Reviewer_316nOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

Motivated by the need to align process reward modeling with downstream selection decisions, the authors develop Subset Champion Marginalization (SCM), a decision-aware training objective for process reward models (PRMs). SCM assigns pairwise comparison weights according to the probability that a correct–incorrect trajectory pair determines the selection outcome under different candidate budgets. The method analytically marginalizes these weights across budgets and combines trajectory-level comparisons with step-level supervision. Experiments on four mathematical reasoning benchmarks demonstrate improvements in Best-of-N selection and PRM-guided beam search across model sizes ranging from 0.6B to 7B.

Critical strengths
  • SCM addresses an important limitation of conventional pairwise supervision by explicitly accounting for candidate availability and the decision relevance of comparisons. The theoretical characterization of selection risk and the closed-form computation of comparison weights provide a principled foundation for the proposed objective.

  • SCM outperforms matched PQM and CRM baselines across multiple model sizes, candidate generators, and inference budgets. The ablation studies further demonstrate the benefits of combining step-level and trajectory-level supervision and adopting decision-aware comparison weights.

Critical weaknesses
  • SCM derives its comparison weights under a score-independent, with-replacement candidate sampling distribution conditioned on mixed candidate sets. However, the main Best-of-N evaluation uses ordered candidate prefixes, while beam search adaptively generates candidates based on PRM scores. Although Appendix B.5 examines alternative availability distributions, it does not investigate their effects during training. Consequently, the extent to which the theoretical formulation explains the observed improvements in practical settings remains unclear.

  • Although SCM is designed for process reward modeling, its evaluation primarily focuses on trajectory selection through Best-of-N and beam search. The paper lacks direct assessments of intermediate-step correctness, such as error localization or step-level reward calibration. Consequently, it remains unclear whether SCM improves the quality of process-level supervision or primarily enhances trajectory-level ranking.

Questions for authors
  • How sensitive is SCM to the assumed candidate-availability distribution and budget prior? Could the authors compare models trained with different availability laws under otherwise identical settings, rather than only evaluating fixed models under different sampling protocols?

  • What is the additional training overhead of SCM compared with PQM and CRM in terms of computation time and memory consumption? How does this overhead scale with the number of candidate trajectories per question, and does SCM remain practical for substantially larger candidate pools?

  • Does SCM improve intermediate-step error detection on benchmarks such as ProcessBench or PRMBench? This evaluation would help determine whether the improvements extend beyond trajectory-level selection to process-reward quality itself.

5 comments
AuthorsOpenReview ↗

Response to Reviewer 316n (Part I)

We appreciate your careful reading and constructive feedback. We are deeply encouraged the reviewer recognized the strengths of our work from various perspectives. Below we respond to your comments and questions in detail.

Weakness 1: However, the main Best-of-N evaluation uses ordered candidate prefixes, while beam search adaptively generates candidates based on PRM scores.

Although Appendix B.5 examines alternative availability distributions, it does not investigate their effects during training.

Question 1: How sensitive is SCM to the assumed candidate-availability distribution and budget prior?

Thank you for distinguishing the theoretical assumptions from their practical use. Section 3.1 derives selection risk for any specified, score-independent candidate law; WR is a closed-form instance. This connects reward learning to downstream decisions, as in predict-then-optimize methods [1].

For a fixed scorer, μij\mu_{ij} is the probability that correct trajectory ii and incorrect trajectory jj are their class champions, conditional on both classes being available. Partitioning collections by this pair gives

J_\mu(S)=\sum_{i,j}\mu_{ij}\phi_\tau(S_j^- - S_i^+),$$ where $e_{ij}$ indicates error under fixed tie-breaking and $\phi_\tau(g)=\tau\log(1+\exp(g/\tau))$. Section 3.2 identifies the unique linear risk weights for fixed within-class orders; Appendix A.4 extends the construction to nonuniform sampling. Surrogate bounds connect optimization to decision risk [2]. For two laws on the same pool, conditioned on mixedness with common within-class orders, Section 3.3 gives $$R_\nu(S)\le R_\mu(S)+d_{\mathrm{sel}}(\mu,\nu) \le\frac{J_\mu(S)-\phi_\tau(-1)}{\phi_\tau(0)-\phi_\tau(-1)}+d_{\mathrm{sel}}(\mu,\nu).$$ Scores lie in $[0,1]$: gaps are at least $-1$, and error requires a nonnegative gap. The discrepancy $d_{\mathrm{sel}}$ is an integral probability metric over realizable error indicators [3], bounded by $\operatorname{TV}=\tfrac12\sum_{ij}|\mu_{ij}-\nu_{ij}|$. This separates surrogate quality from changes in availability. Conditioning isolates ranking from coverage: $\Pr(\mathrm{error})=\Pr(\mathrm{all\\ negative})+\Pr(\mathrm{mixed})R$. Overall accuracy retains all questions and coverage terms. Original Appendix B.5 reports SCM macro AUBCs of 49.26%/48.71%/48.29% under prefix/WOR/WR, ahead of CRM by 2.42–2.54 pp. Section 4.3 tests the same 7B BoN checkpoints in adaptive beam search, with **1.05–3.00 pp gains in all eight configurations** over the stronger PQM/CRM baseline. Earlier pruning changes later inputs, as in sequential prediction [4]. These are direct search results, complementing the fixed-law theory. Your request for training-time comparisons also prompted matched retraining. With group size $K$ and a uniform prior over $N=2,\ldots,K$, WR-U and WOR-U achieve **48.10%/48.41% macro AUBC**, versus PQM/CRM at 46.76%/46.84%. Only the replacement rule changes. Separately, Table 1 varies only the prior over $N=(8,16,32,64,128)$; all alternatives outperform PQM/CRM. These new training-law and prior results are in revised Appendices B.5 and B.7, marked in blue. **Table 1.** Training-budget priors on MATH-500; macro AUBC (%). | Prior | Mixture weights | Macro AUBC (%) | |---|---|---:| | Default | $(1,2,2,2,1)/8$ | **49.26** | | Uniform | $(1,1,1,1,1)/5$ | 48.57 | | Low-budget emphasis | $(16,8,4,2,1)/31$ | 48.32 | | High-budget emphasis | $(1,2,4,8,16)/31$ | 47.73 | **References** [1] Adam N. Elmachtoub and Paul Grigas. Smart “Predict, then Optimize”. *Management Science*, 68(1):9–26, 2022. doi: 10.1287/mnsc.2020.3922. [2] Peter L. Bartlett, Michael I. Jordan, and Jon D. McAuliffe. Convexity, classification, and risk bounds. *Journal of the American Statistical Association*, 101(473):138–156, 2006. doi: 10.1198/016214505000000907. [3] Alfred Müller. Integral probability metrics and their generating classes of functions. *Advances in Applied Probability*, 29(2):429–443, 1997. doi: 10.2307/1428011. [4] Stéphane Ross, Geoffrey Gordon, and Drew Bagnell. [A reduction of imitation learning and structured prediction to no-regret online learning](https://proceedings.mlr.press/v15/ross11a.html). In *Proceedings of the Fourteenth International Conference on Artificial Intelligence and Statistics*, volume 15 of *Proceedings of Machine Learning Research*, pp. 627–635. PMLR, 2011.
AuthorsOpenReview ↗

Response to Reviewer 316n (Part II)

Weakness 2: The paper lacks direct assessments of intermediate-step correctness, such as error localization or step-level reward calibration.

Question 3: Does SCM improve intermediate-step error detection on benchmarks such as ProcessBench or PRMBench?

We agree that trajectory selection and step verification should be evaluated separately. We therefore conducted two complementary evaluations: independent error localization and prediction of the original supervision labels.

Table 2 shows the highest macro F for SCM, 38.92, exceeding CRM/PQM by 2.64/7.56 points. SCM improves AeA_e and F over CRM on all four subsets; CRM retains higher AcA_c on MATH and OlympiadBench. All methods use final checkpoints and official step boundaries.

Table 2. ProcessBench results (%). AcA_c: fully-correct recognition; AeA_e: first-error localization; FF: their harmonic mean.

MethodGSM8K Ac/Ae/FA_c/A_e/FMATH Ac/Ae/FA_c/A_e/FOlympiadBench Ac/Ae/FA_c/A_e/FOmniMath Ac/Ae/FA_c/A_e/FMacro F
PQM87.05 / 58.45 / 69.9445.07 / 26.43 / 33.325.90 / 8.32 / 6.9022.82 / 11.46 / 15.2631.36
CRM94.82 / 45.89 / 61.8587.68 / 29.80 / 44.4844.25 / 13.46 / 20.6447.72 / 11.20 / 18.1436.28
SCM95.34 / 47.34 / 63.2775.86 / 33.00 / 45.9940.12 / 17.55 / 24.4248.55 / 14.23 / 22.0138.92

On the held-out rollout-derived step labels, SCM preserves step-level prediction quality while modestly improving BCE, AUROC, and Brier score over Step-only, as shown in Table 3. These results indicate that its trajectory-selection gains do not come at the expense of predicting the original step-supervision targets. The evaluation covers 108,119 valid steps from 691 held-out groups and 19,176 trajectories, unused for training or checkpoint selection; both models use final step-1554 checkpoints. BCE and Brier are strictly proper measures of overall probability quality [5], while AUROC measures discrimination. Both step-level evaluations are now reported in Appendix B.8, with their protocols in Appendix C.7.

Table 3. Prediction of held-out original step labels.

MethodBCE ↓AUROC ↑Brier ↓
Step-only0.2848090.9399290.087530
SCM0.2820070.9404260.086462

References

[5] Tilmann Gneiting and Adrian E. Raftery. Strictly proper scoring rules, prediction, and estimation. Journal of the American Statistical Association, 102(477):359–378, 2007. doi: 10.1198/016214506000001437.

AuthorsOpenReview ↗

Response to Reviewer 316n (Part III)

Question 2: What is the additional training overhead of SCM compared with PQM and CRM in terms of computation time and memory consumption?

How does this overhead scale with the number of candidate trajectories per question, and does SCM remain practical for substantially larger candidate pools?

The practical value of SCM depends on both its selection gains and its training cost. We measured both complete-update cost and its behavior as candidate pools grow, using four RTX 4090 GPUs, identical real batches, the same rank-64 model, and fully enabled trajectory losses.

Table 4 reports complete-update costs. SCM gains 0.95 pp in AUBC over Uniform pairs for 1.47% additional update time under the same two-pass implementation. Relative to one-pass PQM/CRM, SCM gains 2.50/2.42 pp with 29.2% more update time. Peak memory is equal at the reported precision.

A shorter SCM run using 1,244 updates with the original data and schedule reaches 48.57% macro AUBC, compared with 46.76% for PQM and 46.84% for CRM after full training. Based on the measured per-update times, this corresponds to approximately 1.03 times the update-time cost of full-length PQM/CRM training, while SCM retains higher selection performance.

Table 4. Complete-update cost on four RTX 4090 GPUs.

MethodSeconds/update, mean ± SDTokens/sPeak GiB/GPU
SCM10.771±0.08010.771\pm0.0805,97416.86
Uniform pairs10.615±0.01910.615\pm0.0196,06216.86
PQM8.335±0.0318.335\pm0.0317,71916.86
CRM8.336±0.0218.336\pm0.0217,71916.86

Throughput counts non-padding tokens across all four GPUs. SD is across three timing-round means, each based on eight complete updates after warm-up.

Table 5 fixes eight questions per update and varies K=8K=8–128. SCM stays within 0.2% of Uniform pairs in update time. A 16-fold increase in input takes 14.14-fold more time, while dynamic microbatching keeps memory at 16.74–16.87 GiB/GPU for sequences up to 891 tokens.

The pairwise construction costs O(∣B∣ab)O(|\mathcal B|ab) plus sorting, where a,ba,b count correct and incorrect trajectories and B\mathcal B is the budget grid. These measurements establish practical complete-update cost through K=128K=128. The full timing and scaling results are in Appendix B.10 of the revised manuscript, together with the profiling setup.

Table 5. Training cost as candidates per question increase; eight questions per update.

KKSCM s/updateUniform s/updatePQM s/updateCRM s/updateSCM peak GiB/GPU
85.0695.0793.8963.90516.74
169.6399.6417.3857.37716.84
3218.65618.67614.26314.26816.86
6436.27636.25027.64827.65416.87
12871.66371.58054.56854.58916.87

Comments on the paper

2 comments
AuthorsOpenReview ↗

General Response

We thank Reviewers cuSg, 316n, and 7QBr for their careful reviews and constructive feedback. We are encouraged that all three reviewers recognized SCM’s central contribution of linking process reward model training to downstream selection risk through candidate availability. Reviewer cuSg highlighted the theoretical characterization of the comparison weights as the paper’s strongest contribution. Reviewer 316n noted that the closed-form weights provide a principled basis for the training objective, and reviewer 7QBr emphasized that SCM brings training closer to how reward models are used for solution selection while preserving the inference-time scoring rule. We also appreciate the reviewers’ recognition of the gains over matched baselines and the role of the ablations in separating the effects of different training choices.

In response to the reviewers’ suggestions, we added the following analyses and experiments.

  1. Ranking baselines and weighting gains (cuSg, 7QBr). SCM outperforms matched ListMLE and LambdaRank baselines in all 15 settings. It also improves over Uniform pairs in all three seeds, with a paired macro AUBC gain of 0.81 ± 0.19 percentage points (mean ± sample SD).

  2. Candidate distributions and adaptive search (cuSg, 316n). We clarified that the risk decomposition applies to any specified, score-independent candidate law and added matched retraining with and without replacement and AIME evaluations at three sampling temperatures. The original beam-search results show gains over both matched baselines in all eight 7B verifier settings.

  3. Budget aggregation and training choices (cuSg, 316n, 7QBr). We tested alternative budget priors, off-grid inference budgets, and loss settings. Multi-budget SCM reaches 49.26% macro AUBC versus 47.91% and 47.61% for single-budget training, and leads both alternatives at every evaluation budget on average across the three pools.

  4. Step-level prediction and verification (316n, 7QBr). SCM slightly improves BCE, AUROC, and Brier score over Step-only on held-out original step labels and achieves the highest macro F score among the compared methods on ProcessBench.

  5. Cross-domain results and training cost (cuSg, 316n). SCM leads overall BoN accuracy at all three budgets across four domains beyond mathematics. With the same reported peak memory, SCM takes 1.47% more time per update than Uniform pairs and about 29.2% more than PQM and CRM. We profiled up to 128 candidates per question. At 1,244 updates, SCM still reaches 48.57% macro AUBC, ahead of all four full-length ranking baselines.

We appreciate reviewer 7QBr’s follow-up assessment that the additional experiments addressed all three original concerns. We thank all reviewers again for helping us clarify the contribution and strengthen its empirical support. The separate Summary of Revisions lists where the new results and protocols appear in the manuscript.

AuthorsOpenReview ↗

Summary of Revision

We revised the manuscript in response to the reviewers’ feedback. The main changes are highlighted in blue.

[cuSg, 7QBr] Added matched comparisons with ListMLE and LambdaRank in Section 4.4 and reported three-seed results against Uniform pairs in Appendix B.6.

[cuSg, 316n] Added matched training runs with and without replacement and evaluated newly generated AIME candidates at three temperatures in Appendix B.5.

[cuSg, 316n, 7QBr] Added sensitivity analyses for the loss settings in Section 4.4, along with alternative budget priors, single-budget training, and evaluations at budgets outside the training grid in Appendix B.7.

[cuSg] Added a four-domain evaluation beyond mathematics in Section 4.5.

[316n, 7QBr] Added ProcessBench results and a comparison with Step-only on held-out original step labels in Appendix B.8.

[cuSg, 316n] Added training time, throughput, and memory measurements, candidate-group scaling up to 128 trajectories per question, and a shorter 1,244-update SCM run in Appendix B.10.

We thank the reviewers again for the suggestions that helped us strengthen the manuscript.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.