acceptodds
Under review as a conference paper at ICLR 2027

RRC: Unlocking Generative Reward Models in LLM Reinforcement Learning via Ranking-Based Reward Construction

Abstract

discriminative reward models to generative reward models. However, despite their strong capabilities in response ranking, generative reward models have not realized their potential in reinforcement learning (RL). Our analysis reveals that this limitation arises from a mismatch between the comparative nature of generative reward modeling and the scalar scoring paradigm adopted by existing RL algorithms. To bridge this gap, we propose a Ranking-based Reward Construction (RRC) approach, which enables generative reward models to provide more effective RL learning signals by deriving rewards from relative preference rankings. RRC introduces two complementary strategies: self-competitive ranking, which exploits comparisons among sampled responses, and anchor-guided ranking, which enables scalable ranking-based reward construction with a small set of reference responses. Experiments across open-ended chat and reasoning benchmarks demonstrate that RRC substantially improves RL training with generative reward models, achieving consistent gains over existing reward construction approaches (e.g., 35.8%→41.3% on AlpacaEval2 and 8.0%→11.2% on ArenaHardV2).

Primary area:
generative models
Keywords:
Generative Reward Models, Reinforcement Learning, Response Ranking, Reward Construction

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_yKNkOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper investigates why the ranking capabilities of generative reward models do not consistently translate into better RL training when preference-token probabilities are used as rewards. It proposes Ranking-based Reward Construction (RRC), which converts pairwise judgments into scalar rewards through either comparisons among sampled responses (SCR) or comparisons against reference-policy anchors (AGR). Majority voting and conflict-aware aggregation improve ranking robustness. The resulting rewards are used in GRPO without changing the reward-model training objective. Experiments across chat, knowledge, and math benchmarks report improvements over probability-based reward construction and discriminative reward models.

Critical strengths
  1. The paper is clear and well organized. The motivating comparison, diagrams, and description of the two reward-construction strategies make the overall approach easy to follow. The distinction between preference-token confidence and response quality is useful, particularly when the judge generates reasoning before its final decision.

  2. The empirical improvements are encouraging. RRC improves several benchmarks across two policy backbones and two reward-model sizes. The appendix includes compute-matched probability-aggregation baselines, voting and conflict-adjustment ablations, and training-time comparisons. These provide more support than a comparison against the basic probability baseline alone.

Critical weaknesses
  1. The central margin-awareness claim is not established. Property 2 requires reward differences to be proportional to latent quality differences, but SCR and AGR count binary wins. Even with perfect comparisons, responses with qualities 0, 1, and 100 receive equally spaced SCR rewards. AGR instead measures how many anchors fall below a response, which depends on the anchor distribution and can assign identical rewards to responses of different quality. Appendix A establishes invariance under an assumed affine transformation of oracle quality, but its subsequent discussion does not prove that RRC satisfies this assumption. The empirical benefit may come from useful ordinal signals rather than recovery of quality margins.

  2. The methodological novelty and attribution of the gains remain limited. The main components are win-count aggregation, reference comparisons, majority voting, and an existing ranking heuristic. Their integration is useful, but the paper needs a sharper distinction from prior ranking/reference-based reward construction. The compute-matched baselines in Appendix C.2 are welcome, yet it is unclear whether they compare against the same anchor set and opponent distribution. A matched comparison of hard wins, mean preference probabilities, and calibrated preference logits would better isolate the claimed benefit of ranking over probability-based scoring.

  3. Some important practical evidence and implementation details are missing. No multi-seed variability or uncertainty estimates are reported for the RL results. The reference policy and anchor sampling procedure are insufficiently specified, although anchor quality is central to AGR. Its advertised query savings also do not apply to the main setting: with eight sampled responses and eight anchors, AGR requires 64 pair comparisons versus 28 for SCR at the same voting budget. The favorable wall-clock results are useful, but need a clearer explanation of hardware allocation, caching, and generation costs.

Questions for authors
  1. Under what assumptions do win-count differences approximate latent quality margins? Could you revise the guarantee to distinguish order/rank information from cardinal quality differences, and address the equally spaced SCR rewards in the example above?

  2. How frequently does AGR assign the same reward to every response in a group, producing no preference-learning signal? Does this become more common as the policy improves beyond the fixed anchors, and would quality-diverse or periodically refreshed anchors help?

  3. Could you compare hard-win counts, averaged preference probabilities, and calibrated log-odds using exactly the same comparisons, anchors, reasoning budget, and voting samples? Please clarify the opponent/reference construction used in Table 5.

  4. Which model generates the anchors, with what decoding settings, and are the same anchor responses reused throughout training? Please also provide the number of RL training steps and multi-seed results for the main comparisons.

  5. How do the query counts of 64 for AGR and 28 for SCR in the main setting reconcile with AGR's lower training time? Does Table 4 include anchor precomputation and all reward-serving GPU resources on an equal footing?

  6. Equation (5) assigns rewards from observed wins, whereas Algorithm 1 assigns rewards from positions in a total order, including when conflict adjustment is disabled. Which is implemented, and how are ties handled? Also, please distinguish the optimal Kemeny objective in Property 3 from the greedy heuristic used in practice, which need not maximize weighted agreement.

6 comments
AuthorsOpenReview ↗

Response (1/6)

Dear Reviewer yKNk,

We sincerely appreciate your constructive and thoughtful feedback.

We greatly appreciate your recognition that our paper is "clear and well organized" and that the distinction between preference-token confidence and response quality is "useful". We also appreciate your encouraging assessment of the empirical improvements and your acknowledgment of the additional controls in the appendix.

Below, we address the related weaknesses and questions together.


W1 & Q1: Margin Awareness and Ordinal Information

Thanks for your insightful comment! We would like to clarify the distinction between ordinal information and quality margins.

  1. Appendix A gives a sufficient condition for preserving normalized advantages. It does not prove that win-count rewards satisfy this condition. In your example, qualities (0,1,100)(0,1,100) yield SCR rewards (0,α,2α)(0,\alpha,2\alpha). The order is preserved, but the quality gaps are not. Binary rankings cannot distinguish quality scales related by a strictly increasing transformation.

  2. For AGR, assume noiseless comparisons and independent anchor qualities drawn from a continuous CDF FAF_A. For a fixed prompt,

    E[ri−rj∣qi,qj]=αn(FA(qi)−FA(qj)).\mathbb{E}[r_i-r_j\mid q_i,q_j]=\alpha n\bigl(F_A(q_i)-F_A(q_j)\bigr).

    Expected reward gaps measure percentile differences within the anchor distribution. They are proportional to quality gaps when FAF_A is affine with positive slope over the relevant interval. Nearly constant positive density gives a local approximation. Finite anchors can still produce ties.

RRC can provide useful ordinal learning signals without recovering oracle margins.

We promise to revise this claim and add this discussion in the revision.

AuthorsOpenReview ↗

Response (2/6)

W2 & Q3: Novelty and Comparisons with Identical Judgments

Thanks for your helpful feedback! We would like to clarify our contribution and separate the existing strengthened PRC baselines from the additional shared-judgment experiment.

  1. We build on established ranking methods. RRHF [1] aligns policy log-probabilities with preferences through a ranking loss. RRC converts a fixed judge's comparisons into scalar rewards for GRPO. Our contribution concerns how comparative judgments become policy-learning signals. We view reference comparisons and win-count aggregation as established techniques that support this construction.

  2. Appendix C.2, Table 6 in the revised paper (Table 5 in the original submission) compares RRC with strengthened PRC baselines. On AE2, PRC with majority voting and soft aggregation score 32.8 and 30.1, compared with 36.4 for SCR and 35.8 for AGR. At the main group sizes, SCR uses 28 comparisons per vote and AGR uses 64. Please note that this table alone does not establish a shared query budget or identical opponents and judgment samples. To address your concern, the shared-judgment comparison below directly controls the comparison inputs.

  3. Comparison with identical judgments. The additional offline comparison uses 100 prompts, eight responses and eight fixed anchors per prompt, and four judge samples per pair: 25,600 judgments in total. All methods reuse the same pairs, A/B assignments, reasoning traces, and token probabilities. The reasoning cap is 512 tokens.

    Hard wins use a strict majority, with no win for a 2-2 split. Mean probabilities average judge probabilities before summing over anchors. Calibrated log-odds use Platt scaling fitted on 2,000 disjoint preference pairs, then average logits over samples and sum over anchors. Probabilities are normalized over the two preference labels and clipped to [10−6,1−10−6][10^{-6},1-10^{-6}] for logit computation.

    Reward ConstructionAgreement with Model-Generated Ordering (%)Zero-Variance Groups (%)AE2 after Short RL
    Hard-win aggregation84.68032.7
    Mean preference probabilities91.11031.9
    Calibrated log-odds92.43033.1

    In both offline comparisons (here and in Q6), the reference orderings are generated by the same model being trained. Agreement measures consistency with this model-generated reference, rather than accuracy against independent human annotations. We give half credit for tied predictions. Zero variance is measured before adding the format reward. The short RL runs use 50 optimizer steps with the same initial policy, anchors, and judge budget. Policy responses can diverge after updates, so identical judgment samples apply to the offline comparison only.

    The results show that:

    • Calibrated log-odds achieves the highest offline agreement and short-run AE2 in this comparison. Hard wins score 0.8 points above mean probabilities on AE2, but 0.4 points below calibrated log-odds. These results do not establish that hard wins are uniformly better than probability-based rewards.

    • The log-odds result does not isolate a benefit from calibration. Without clipping saturation, a shared positive Platt transform preserves the aggregated ranking. It also cancels under ideal group normalization of the ranking reward alone. Format rewards, normalization epsilon, clipping, and reward scaling can affect the RL update.

    • None of the 100 groups has identical rewards under any of the three methods. This describes the evaluated sample, not the population rate. The AGR results in Q2 use a different sample and voting setting.

We promise to add these experiments and this discussion in the revision.

Reference

[1] Zheng Yuan et al. RRHF: Rank Responses to Align Language Models with Human Feedback without Tears. NeurIPS 2023.

AuthorsOpenReview ↗

Response (3/6)

Q2: Reward Ties and Anchor Coverage

Thanks for this important question! We would like to clarify that reference-policy anchors keep the comparison distribution fixed. Their quality coverage may still become limited as training progresses.

  1. Anchor coverage. The 3B-scale, w/ Thinking comparison uses eight anchors without additional voting.

    Anchor DistributionAE2Zero-Variance Groups (%)All-Win Responses (%)
    Reference-policy anchors35.8822
    Narrow-quality anchors33.41936
    Anchors spanning quality levels36.3513

    Zero-variance groups have identical ranking rewards before the format reward. All-win responses defeat every anchor.

    • Broader anchor coverage reduces reward ties from 8% to 5% and all-win responses from 22% to 13%. AE2 increases from 35.8 to 36.3.
    • Tied groups provide no ranking-based advantage, although format rewards may still differ.
  2. Changes during training. We evaluate equal-size rollout samples on common prompts at 20%, 50%, and 90% of training. The refreshed strategy generates eight new anchors per prompt from the policy checkpoints at 40% and 70%.

    Anchor StrategyZero-Variance Groups: Early / Middle / Late (%)All-Win Responses: Early / Middle / Late (%)Final AE2
    Fixed reference anchors5 / 7 / 1212 / 20 / 3435.8
    Fixed quality-diverse anchors4 / 6 / 58 / 12 / 1936.3
    Periodically refreshed anchors5 / 4 / 612 / 10 / 1636.1
    • Fixed reference anchors lose discriminative power in this comparison. Their tie rate rises from 5% to 12%, and the all-win rate rises from 12% to 34%.

    • Both diverse and refreshed anchors reduce late saturation. Refreshing achieves AE2 of 36.1, compared with 35.8 for fixed reference anchors and 36.3 for fixed diverse anchors. Refreshing helps in this run, but does not outperform fixed diverse anchors.

    The fixed-anchor aggregate rates above equal the averages over these three equally sized evaluation windows. They describe anchor coverage across training, rather than the second-half optimization diagnostics.

We promise to add these experiments in the revision.

Reviewer_yV1jOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes Ranking-based Reward Construction (RRC), a framework designed to better utilize generative reward models (GRMs) in reinforcement learning for large language models. The authors argue that existing probability-based reward construction methods fail to fully exploit the comparative capabilities of GRMs, resulting in a mismatch between preference ranking and scalar reward optimization. To address this issue, RRC introduces two complementary strategies: Self-Competitive Ranking (SCR), which constructs rewards from pairwise comparisons among sampled responses, and Anchor-Guided Ranking (AGR), which evaluates responses against a set of reference anchors to reduce comparison costs. The framework further incorporates majority voting and conflict-aware ranking adjustment to improve the reliability of preference judgments. Experiments on open-ended dialogue and reasoning benchmarks demonstrate consistent improvements over discriminative reward models and probability-based generative reward construction methods.

Critical strengths
  1. The paper identifies a relevant limitation in applying generative reward models to reinforcement learning. The observation that strong pairwise preference ranking performance does not necessarily translate into effective scalar rewards for policy optimization is meaningful. The proposed framework provides a practical way to exploit the comparative capabilities of generative reward models rather than relying exclusively on preference-token probabilities.

  2. The proposed framework is straightforward and compatible with existing reinforcement learning algorithms. Both SCR and AGR provide relatively simple mechanisms for converting preference judgments into numerical rewards. In particular, the anchor-guided strategy offers a practical alternative to exhaustive pairwise comparisons, reducing the number of comparisons from quadratic to linear in the number of sampled responses when the anchor count is fixed.

Critical weaknesses
  1. The methodological novelty is relatively limited, as the central components largely rely on established ranking and aggregation techniques. The proposed SCR assigns reward scores according to the number of pairwise wins, which is closely related to conventional tournament-based ranking and Borda-style scoring methods. Similarly, AGR estimates response quality through comparisons against reference responses, an intuitive extension of anchor-based ranking and relative evaluation. Majority voting and Kemeny-inspired conflict resolution are also well-established techniques. Although integrating these mechanisms into GRM-based reinforcement learning is practically useful, the paper does not clearly establish what fundamentally new algorithmic principle is introduced beyond adapting existing ranking methods to reward construction.
  2. The theoretical justification for margin-aware reward construction is insufficient. A central claim of the paper is that ranking-based rewards satisfy both order preservation and margin awareness. However, the proposed reward functions are based on discrete pairwise win counts, which do not necessarily reflect the underlying quality differences between responses. For example, two responses with substantially different latent qualities may obtain identical rewards if they defeat the same set of anchors. Conversely, small differences in quality may produce large differences in win counts depending on the distribution of competing responses or anchors. The analysis in Appendix A establishes that rewards related to oracle quality through a positive affine transformation preserve normalized advantages, but it does not demonstrate that the proposed win-count rewards satisfy this condition. Consequently, the theoretical analysis does not fully support the claimed relationship between ranking-based reward differences and oracle preference margins.
  3. The experimental comparisons do not completely isolate the benefits of the proposed reward construction mechanism. The generative reward models are trained with textual rationales from HelpSteer3, whereas the discriminative reward models use only binary preference labels. Although the models share the same preference examples, this introduces a difference in supervision that complicates attributing performance improvements solely to the reward modeling paradigm. Moreover, RRC introduces additional preference comparisons, majority voting, and ranking adjustment, making it important to distinguish improvements from better reward construction versus increased inference-time computation.
  4. The robustness and generalizability of the proposed reward construction strategies require further investigation. Both SCR and AGR depend heavily on the reliability of pairwise preference judgments. However, generative reward models may exhibit position bias, inconsistent preferences, or systematic errors, which can directly affect the resulting reward rankings. Although majority voting and conflict-aware adjustment are introduced to mitigate these issues, the paper does not sufficiently investigate how different levels and types of preference noise influence reward quality and downstream policy optimization. In addition, AGR depends on the quality and diversity of anchor responses generated by a reference policy. If the anchors are concentrated within a narrow quality range or become unrepresentative as the policy improves, the resulting reward scores may lose discriminative power.
  5. The empirical evaluation lacks sufficient statistical evidence, and the relationship between ranking-based rewards and optimization dynamics remains underexplored. Although RRC reports consistent improvements across several benchmarks, the main experiments do not provide multiple training-seed results or statistical significance analyses. This makes it difficult to assess the reliability of some relatively modest gains, particularly on reasoning benchmarks. Furthermore, the paper primarily evaluates final downstream performance without sufficiently examining how the constructed rewards influence policy optimization.
Questions for authors

See Weaknesses

3 comments
AuthorsOpenReview ↗

Response (1/3)

Dear Reviewer yV1j,

We sincerely appreciate your constructive and thoughtful feedback.

We greatly appreciate your recognition that our paper identifies a "relevant limitation" in applying GRMs to RL and provides a "practical way to exploit the comparative capabilities of generative reward models". We also appreciate your acknowledgment that our framework is "straightforward and compatible with existing reinforcement learning algorithms", as well as AGR's efficiency in reducing comparisons "from quadratic to linear".

Below, we would like to clarify the main points you raised.


W1: Ranking-Based Reward Construction Mechanisms

Thanks for your insightful comment! As you noted, the ranking techniques are established. We would like to clarify how we use them for GRM-based reward construction. Our contribution lies in using GRMs' comparative judgments to construct effective rewards for policy optimization.

Figure 1 shows that the GRM improves ranking accuracy from 75.1 to 79.8 on RM-Bench and from 74.7 to 77.2 on JudgeBench. Yet peak AE2 performance during RL increases only from 34.5 to 35.8. This gap motivates our study of reward construction. SCR derives rewards from comparisons among current-policy responses. AGR uses reference-policy anchors to scale this construction to larger response groups.

With the same 3B GRM under the w/ Thinking setting, SCR and AGR improve AE2 from 31.4 to 36.4 and 35.8, respectively (Table 1). Neither result uses additional voting. The strengthened PRC baselines in W3 provide an additional comparison of reward construction.

We promise to add this discussion in the revision.

W2: Theoretical Justification for Margin-Aware Rewards

Thanks for your insightful comment! We would like to clarify the assumptions and scope of the analysis: Appendix A gives a sufficient condition for preserving normalized advantages. It does not prove that SCR or AGR recovers oracle quality margins.

For a fixed prompt, assume noiseless comparisons and independent anchor qualities drawn from a continuous CDF FAF_A. AGR then satisfies

E[ri−rj∣qi,qj]=αn(FA(qi)−FA(qj)).\mathbb{E}[r_i-r_j\mid q_i,q_j]=\alpha n\bigl(F_A(q_i)-F_A(q_j)\bigr).

Thus, expected AGR reward gaps measure percentile differences within the anchor distribution. They are proportional to quality gaps when FAF_A is affine with positive slope over the relevant interval. Nearly constant positive density gives a local approximation. SCR likewise measures relative standing among current-policy responses.

As you noted, finite anchors can assign equal rewards to responses with different qualities. Uneven anchor density can also distort reward gaps. Win-count rewards therefore do not guarantee proportionality to oracle margins.

We promise to revise this claim and add this discussion in the revision.

AuthorsOpenReview ↗

Response (2/3)

W3: Fairness of Experimental Comparisons

Thanks for your helpful feedback! We would like to distinguish the motivation study from the main PRC-versus-RRC comparison and clarify the controls used in each.

  1. Figure 1 examines whether stronger ranking performance translates into better RL performance. It does not isolate model architecture, since only the GRM receives rationales. Please note that the main PRC-versus-RRC comparison uses the same trained GRM. Reward-model supervision is therefore held fixed in this comparison.

  2. We would also like to clarify the comparison in Appendix C.2, Table 6 in the revised paper (Table 5 in the original submission). It evaluates RRC against strengthened PRC baselines using 3B reward models with Thinking.

    Reward ConstructionAE2WiBMMRMATH
    PRC + Majority Voting32.852.152.845.2
    PRC + Soft Aggregation30.151.448.244.8
    RRC-SCR36.455.654.445.6
    RRC-AGR35.856.855.046.4

    The results show that:

    • RRC outperforms both PRC baselines on all four benchmarks. This comparison supports the use of ranking-based reward construction.
    • These results do not isolate aggregation under identical comparisons. With eight responses and eight anchors, SCR uses 28 comparisons per vote and AGR uses 64. These counts alone do not establish a shared query budget across methods.
    • AGR also improves AE2 at nearly the same training time. Table 5 in the revised paper (Table 4 in the original submission) reports an increase from 35.8 to 39.4 with an 8B GRM and Thinking, at 10.6h versus 10.5h for PRC.

We promise to add this discussion in the revision.

W4: Robustness to Preference Noise and Anchor Quality

Thanks for the thoughtful suggestion! To address your concern, we summarize three relevant analyses from the paper below.

  • Appendix B.1 describes our random assignment of responses to A/B positions. Figure 1(a) shows that voting@8 improves RM-Bench from 79.8 to 82.6 and JudgeBench from 77.2 to 78.9. Voting improves ranking accuracy, though it cannot reliably correct systematic errors.

  • Appendix B.2, Figure 4(b) reports the CARA ablation. CARA improves AE2 by 0.6/1.9 points and MMR by 1.3/1.5 points for 3B/8B reward models. CARA improves downstream performance in this ablation.

  • Appendix A, Proposition 1 explains how reference-policy anchors decouple the comparison distribution from policy updates. Section 6.1, Figure 3 shows gains from increasing anchor counts, followed by saturation or slight degradation at 256 anchors. A fixed reference policy does not guarantee adequate anchor coverage throughout training.

We further evaluate preference noise and anchor coverage with the 3B GRM under the w/ Thinking setting.

  1. Preference noise. We inject independent random flips before preference aggregation. Both SCR settings use CARA. Entries report AE2 scores.

    Preference NoiseSCR + CARASCR + voting@8 + CARAAGRAGR + voting@8
    No injected noise36.437.835.837.5
    10% random flips34.737.134.936.9
    20% random flips32.635.833.435.4
    30% random flips30.833.631.734.1
    • Voting reduces the performance drop under random flips. At 20% noise, SCR loses 3.8 points without additional voting and 2.0 points with voting@8. AGR loses 2.4 points without voting and 2.1 points with voting.
    • The SCR comparison holds CARA fixed and evaluates the effect of voting. It does not isolate the effect of CARA. Random flips also do not test systematic position bias.
  2. Anchor coverage. We compare anchor distributions using eight anchors per prompt without additional voting. The tie and all-win rates below average equally sized rollout samples at 20%, 50%, and 90% of training.

    Anchor DistributionAE2Zero-Variance Groups (%)All-Win Responses (%)
    Reference-policy anchors35.8822
    Narrow-quality anchors33.41936
    Anchors spanning quality levels36.3513

    Zero-variance groups have identical ranking rewards before adding the format reward. All-win responses defeat all eight anchors.

    • Broader anchor coverage reduces reward ties and saturation. It also improves AE2 from 35.8 to 36.3.
    • Narrow-quality anchors reduce AE2 to 33.4. These results support the importance of anchor coverage, but do not establish robustness to long-term policy drift.

We promise to add these experiments in the revision.

AuthorsOpenReview ↗

Response (3/3)

W5: Statistical Reliability and Optimization Dynamics

Thanks for this important suggestion! To address your concern, we report multi-seed results and optimization diagnostics below.

  1. Multi-seed results. We compare PRC, SCR, and AGR across five matched training seeds with the 3B GRM under the w/ Thinking setting. The table reports mean and sample standard deviation. MATH scores use one binary evaluation per item on MATH-500.

    Reward ConstructionAE2AH2WiBMMRMATH
    PRC31.40 ± 1.057.84 ± 0.3950.20 ± 1.0052.40 ± 0.6345.32 ± 0.77
    RRC-SCR36.38 ± 0.768.80 ± 0.3855.60 ± 0.9354.36 ± 0.6145.44 ± 1.57
    RRC-AGR35.66 ± 0.749.80 ± 0.5056.70 ± 0.7555.00 ± 0.6146.24 ± 1.05

    Scores for each training seed.

    SeedReward ConstructionAE2AH2WiBMMRMATH
    1PRC30.67.449.451.944.4
    1RRC-SCR35.78.454.854.446.2
    1RRC-AGR35.49.455.654.546.4
    2PRC32.28.250.952.846.2
    2RRC-SCR37.59.456.155.046.6
    2RRC-AGR36.710.557.155.647.0
    3PRC31.37.749.152.644.8
    3RRC-SCR36.18.955.153.945.2
    3RRC-AGR34.89.756.454.446.6
    4PRC32.78.351.553.145.2
    4RRC-SCR36.88.657.054.946.4
    4RRC-AGR36.110.157.655.746.8
    5PRC30.27.650.151.646.0
    5RRC-SCR35.88.755.053.642.8
    5RRC-AGR35.39.356.854.844.4

    The following table reports paired mean differences and two-sided 95% t confidence intervals across the five seeds. Each interval is computed separately, without adjustment for multiple comparisons.

    ComparisonAE2AH2WiBMMRMATH
    RRC-SCR − PRC+4.98 [+4.27, +5.69]+0.96 [+0.49, +1.43]+5.40 [+4.90, +5.90]+1.96 [+1.40, +2.52]+0.12 [-2.30, +2.54]
    RRC-AGR − PRC+4.26 [+3.30, +5.22]+1.96 [+1.67, +2.25]+6.50 [+5.87, +7.13]+2.60 [+1.97, +3.23]+0.92 [-0.92, +2.76]
    • The intervals exclude zero on AE2, AH2, WiB, and MMR for both RRC variants.
    • The MATH gains are less conclusive. Both intervals include zero. At Seed 5, SCR and AGR score below PRC on MATH (42.8 and 44.4 versus 46.0). We retain all five seeds in the analysis.
  2. Optimization dynamics. We examine optimization at common training steps. These diagnostics summarize the second half of training. Their reporting window differs from the three-checkpoint anchor analysis in W4. We log diagnostics every 100 steps and average over checkpoints in the second half of training, including the final checkpoint. We compute reward variance before adding the format reward and use a threshold of 10−810^{-8}. KL is measured per generated token against the frozen reference policy. Clipping fraction measures how often the clipped GRPO term is active over valid generated tokens.

    The SCR zero-variance rate follows analytically from Algorithm 1's total-order reward assignment for complete groups. The other entries are measured.

    Reward ConstructionNear-Zero-Variance Groups (%)KL DivergenceClipping Fraction (%)
    PRC130.0358.1
    RRC-SCR00.0388.7
    RRC-AGR80.0347.9
    • AGR reduces near-zero-variance groups from 13% to 8%. Under Algorithm 1's total-order assignment, complete SCR groups of eight receive rewards 0,0.1,…,0.70, 0.1, \ldots, 0.7, with population variance 0.0525.
    • KL and clipping values remain close to PRC. These diagnostics describe the reward signal and policy updates. They do not establish a causal explanation for the performance gains.

We promise to add these experiments and analyses in the revision.


We sincerely thank you again for your constructive feedback and your time!

Best,

Authors

Reviewer_UYVbOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper points out a mismatch between the comparative judgments of generative reward models (GRMs) and the scalar scores required by existing RL algorithms. GRMs are strong at response ranking, but that advantage is largely lost when the preference-token probability is used as the scalar reward. To address this problem, it proposes Ranking-based Reward Construction (RRC), which builds scalar rewards from relative orderings rather than token probabilities. RRC has two mechanisms. Self-competitive ranking (SCR) scores each response by how many of the other responses sampled for the same prompt it beats. Anchor-guided ranking (AGR) scores each response by how many reference-policy anchors it beats, so the number of comparisons scales linearly with the number of anchors for a fixed set of responses. Results on open-ended chat, knowledge, and math reasoning benchmarks show that RRC consistently improves GRM-based RL training over probability-based and discriminative baselines across multiple reward-model scales. Additional studies examine comparison budgets, anchor counts, ablations, and training cost.

Critical strengths

S1: The method is plug-and-play and opens new scaling dimensions for RL.

RRC requires no retraining of reward model. It works as an inference-time change on the reward computing stage and plugs directly into GRPO. AGR's cost is almost the same as the baseline (10.6h vs 10.5h). The voting budget and the number of anchors act as two new test-time scaling axes (Figure 3).

Critical weaknesses

W1: In Figure 1, the GRM is trained with human-written rationales while the DRM only gets binary labels, so the ranking gap cannot be cleanly attributed to architecture.

W2: The evaluation omits a control that applies the same rank-based reward construction to a discriminative reward model. If that construction also improves DRM-based RL, the gains indicate a general effect of rank-based aggregation, and the claim that RRC specifically unlocks generative reward models is not supported.

Questions for authors

Q1: The paper's critique of scalar scoring is entirely based on one extraction method: preference-token probabilities. A rubrics-based GRM — one that generates rubric items, scores each response item by item, and outputs parsed numeric scores — bypasses probability collapse and can carry margin information in its score differences. This variant is not discussed or tested in the paper. Does the mismatch argument still hold if scalar rewards come from parsed rubric scores instead of token probabilities? If not, the claim should be explicitly limited to probability-based reward construction.

2 comments
AuthorsOpenReview ↗

Response (1/2)

Dear Reviewer UYVb,

We sincerely appreciate your constructive and thoughtful feedback.

We greatly appreciate your recognition that RRC is "plug-and-play and opens new scaling dimensions for RL", as well as its practical efficiency with "almost the same" training cost as the baseline.

Below, we would like to clarify the main points you raised.


W1: Different Supervision Between DRM and GRM

Thanks for your insightful comment! We would like to clarify the purpose of this comparison.

  1. Figure 1 examines whether the ranking advantages of reasoning-augmented GRMs translate into RL improvements. Only the GRM receives human-written rationales, so this comparison does not isolate model architecture. Please note that our main experiments compare PRC and RRC using the same trained GRM. The gains in this comparison reflect the choice of reward construction, rather than differences in reward-model supervision.

  2. To address the supervision difference directly, we train an additional GRM with the same preference labels and training settings as the DRM. It generates preference labels without intermediate reasoning. The DRM and GRM w/o Rationales use matched preference-label supervision. We include the GRM w/ Rationales as a reference.

    Reward ModelRM-BenchJudgeBenchRL (AE2)
    DRM75.174.734.5
    GRM w/o Rationales78.676.234.2
    GRM w/ Rationales79.877.235.8

    The results provide two observations:

    • Without rationales, the GRM achieves higher accuracy than the DRM on both ranking benchmarks, while obtaining similar RL scores. Thus, access to rationales does not fully explain the ranking advantage observed here.
    • With rationales, the GRM further improves ranking accuracy, but yields only modest RL gains. This supports our motivation that stronger ranking capabilities do not necessarily translate into better RL performance under probability-based reward construction.

We promise to add these results and this discussion in the revision.

AuthorsOpenReview ↗

Response (2/2)

W2: Missing DRM Baseline with RRC

Thanks for your helpful feedback! We would like to clarify that RRC can also be applied to DRMs. We focus on GRMs because of their stronger ranking performance, while the same construction also applies to DRMs.

  1. RRC derives rewards from response rankings. As Figure 1 shows, GRMs outperform DRMs on RM-Bench and JudgeBench. Voting further improves their accuracy. These results motivate our use of RRC to translate GRMs' ranking strengths into RL gains. We reproduce the results below for clarity.

    Reward ModelRM-BenchJudgeBench
    DRM75.174.7
    GRM79.877.2
    GRM + voting@480.778.4
    GRM + voting@882.678.9
  2. We also apply RRC to a DRM and compare it with GRM-based RRC under the same RL settings. Both reward models use the 3B scale, under the w/o Thinking setting. We measure gains over direct scalar scoring for the DRM and PRC for the GRM.

    Reward ModelAE2AH2WiBMMRMATH
    DRM27.67.246.350.240.4
    GRM w/ PRC28.46.848.151.741.2
    DRM + RRC-SCR28.88.047.551.140.8
    DRM + RRC-AGR29.17.048.249.841.2
    GRM + RRC-SCR33.18.253.052.442.0
    GRM + RRC-AGR32.88.852.952.745.4

    From the results, we observe that:

    • Benefits extend to DRMs. DRM + RRC-SCR improves over the original DRM on all five benchmarks. This shows that rank-based reward construction is also useful for discriminative models.
    • GRMs generally obtain larger gains. For example, SCR improves AE2 by 4.7 points for the GRM, compared with 1.2 points for the DRM. The corresponding gains on WiB are 4.9 and 1.2 points. These comparisons support the practical value of RRC for GRM-based RL.
    • DRM + RRC-AGR shows small decreases on AH2 and MMR. The gains are not uniform across benchmarks. We therefore interpret these differences as specific to the evaluated settings.

    Taken together, these results suggest that RRC benefits both model types, with larger gains for GRMs in this comparison.

We promise to add this comparison in the revision.

Q1: Generalization to Rubric-Based Reward Scoring

Thanks for your insightful comment! To address your concern, we would like to clarify the scope of the mismatch analysis and provide a direct comparison with rubric-based scoring.

  1. We would like to clarify that our mismatch analysis primarily concerns probability-based reward construction (PRC). Preference-token probabilities can saturate and lose resolution as scalar rewards. Rubric-based scoring directly generates numerical scores and avoids this particular issue. Reward differences still affect group-relative advantages, which motivates examining the quality of rubric scores. Our analysis does not imply that rubric-based rewards necessarily suffer from the same mismatch.

  2. To address your question, we train a rubric-based GRM on the same HelpSteer3 samples and backbone. We derive rubric-based supervision from the preference pairs. The model generates evaluation criteria and numerical scores, which we aggregate into RL rewards. The results below use the 3B-scale, w/o Thinking setting.

    Reward ConstructionAE2AH2WiBMMRMATH
    GRM w/ PRC28.46.848.151.741.2
    Rubric-based GRM30.67.449.952.141.0
    GRM w/ RRC-SCR33.18.253.052.442.0
    GRM w/ RRC-AGR32.88.852.952.745.4

    The results show that:

    • Rubric-based scoring is a useful alternative to PRC. It improves performance on four of the five benchmarks, including AE2 and WiB, while MATH is slightly lower. This supports evaluating numerical scoring separately from preference-token probabilities.
    • Both SCR and AGR outperform this rubric-based baseline on all five benchmarks. RRC provides further gains over the rubric implementation tested here.

We promise to add these results in the revision.


We sincerely thank you for your positive feedback on our paper and for your time!

Best,

Authors

Comments on the paper

4 comments
AuthorsOpenReview ↗

General Response to Reviewers

We sincerely thank all reviewers for their careful reading and constructive feedback.

We appreciate Reviewer UYVb's recognition that RRC is "plug-and-play and opens new scaling dimensions for RL". Reviewer yV1j highlighted the "relevant limitation" identified in our study. We also thank Reviewer yKNk for describing the paper as "clear and well organized" and the empirical improvements as "encouraging".

Below, we summarize our responses to the main concerns. We provide further details in the individual replies.

  1. Reviewer UYVb raised concerns about supervision differences and alternative reward models. We provide a comparison under matched preference-label supervision. We also evaluate DRM-based RRC and rubric-based GRM scoring. These results show that RRC also benefits DRMs and outperforms the rubric baseline tested here.

  2. Reviewer yV1j asked about novelty, theoretical support, and experimental reliability. We clarify our contribution and the assumptions behind margin awareness. We evaluate preference noise and anchor coverage. We also report five-seed results and optimization diagnostics to examine the reliability of the gains and the resulting training behavior.

  3. Reviewer yKNk requested more controlled comparisons and clarification of the reward construction. We compare aggregation methods using identical offline judgments and provide a separate short-RL comparison. We distinguish observed wins from total-order rewards and clarify the scope of the Kemeny result. We also examine reward ties and explain the accounting of query counts and training costs.

We have also uploaded a revised manuscript, with revisions highlighted in blue. The main changes are as follows:

  • "Section 6.2 and Appendix D.1:" We add reward-model baselines and a comparison under matched preference-label supervision.
  • "Section 4 and Appendix A:" We clarify the theoretical assumptions and distinguish the two SCR reward assignments.
  • "Appendix D.2:" We add aggregation comparisons using identical offline judgments and a separate short-RL experiment.
  • "Appendices D.3–D.5:" We add robustness experiments, five-seed results, and optimization diagnostics.
  • "Appendices C.1–C.2:" We clarify query complexity and training costs, including anchor precomputation.

We sincerely thank all reviewers again for their time and helpful suggestions.

Dear Reviewers,

As the discussion period is coming to an end, I kindly encourage you to review and respond to the authors' rebuttal as soon as possible.

Thank you very much for your time and valuable contributions to the review process.

Best regards,
Area Chair

Thank you for the detailed response and additional experiments, which address my main concerns and clarify the scope of the paper’s claims. I will maintain my original score.

AuthorsOpenReview ↗

Thank You for Your Feedback

Dear Reviewer UYVb,

Thank you for your reply. We are glad that our additional experiments and clarifications addressed your main concerns. We sincerely appreciate your time and constructive feedback.

Best,

Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.