- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper investigates why the ranking capabilities of generative reward models do not consistently translate into better RL training when preference-token probabilities are used as rewards. It proposes Ranking-based Reward Construction (RRC), which converts pairwise judgments into scalar rewards through either comparisons among sampled responses (SCR) or comparisons against reference-policy anchors (AGR). Majority voting and conflict-aware aggregation improve ranking robustness. The resulting rewards are used in GRPO without changing the reward-model training objective. Experiments across chat, knowledge, and math benchmarks report improvements over probability-based reward construction and discriminative reward models.
Critical strengths
-
The paper is clear and well organized. The motivating comparison, diagrams, and description of the two reward-construction strategies make the overall approach easy to follow. The distinction between preference-token confidence and response quality is useful, particularly when the judge generates reasoning before its final decision.
-
The empirical improvements are encouraging. RRC improves several benchmarks across two policy backbones and two reward-model sizes. The appendix includes compute-matched probability-aggregation baselines, voting and conflict-adjustment ablations, and training-time comparisons. These provide more support than a comparison against the basic probability baseline alone.
Critical weaknesses
-
The central margin-awareness claim is not established. Property 2 requires reward differences to be proportional to latent quality differences, but SCR and AGR count binary wins. Even with perfect comparisons, responses with qualities 0, 1, and 100 receive equally spaced SCR rewards. AGR instead measures how many anchors fall below a response, which depends on the anchor distribution and can assign identical rewards to responses of different quality. Appendix A establishes invariance under an assumed affine transformation of oracle quality, but its subsequent discussion does not prove that RRC satisfies this assumption. The empirical benefit may come from useful ordinal signals rather than recovery of quality margins.
-
The methodological novelty and attribution of the gains remain limited. The main components are win-count aggregation, reference comparisons, majority voting, and an existing ranking heuristic. Their integration is useful, but the paper needs a sharper distinction from prior ranking/reference-based reward construction. The compute-matched baselines in Appendix C.2 are welcome, yet it is unclear whether they compare against the same anchor set and opponent distribution. A matched comparison of hard wins, mean preference probabilities, and calibrated preference logits would better isolate the claimed benefit of ranking over probability-based scoring.
-
Some important practical evidence and implementation details are missing. No multi-seed variability or uncertainty estimates are reported for the RL results. The reference policy and anchor sampling procedure are insufficiently specified, although anchor quality is central to AGR. Its advertised query savings also do not apply to the main setting: with eight sampled responses and eight anchors, AGR requires 64 pair comparisons versus 28 for SCR at the same voting budget. The favorable wall-clock results are useful, but need a clearer explanation of hardware allocation, caching, and generation costs.
Questions for authors
-
Under what assumptions do win-count differences approximate latent quality margins? Could you revise the guarantee to distinguish order/rank information from cardinal quality differences, and address the equally spaced SCR rewards in the example above?
-
How frequently does AGR assign the same reward to every response in a group, producing no preference-learning signal? Does this become more common as the policy improves beyond the fixed anchors, and would quality-diverse or periodically refreshed anchors help?
-
Could you compare hard-win counts, averaged preference probabilities, and calibrated log-odds using exactly the same comparisons, anchors, reasoning budget, and voting samples? Please clarify the opponent/reference construction used in Table 5.
-
Which model generates the anchors, with what decoding settings, and are the same anchor responses reused throughout training? Please also provide the number of RL training steps and multi-seed results for the main comparisons.
-
How do the query counts of 64 for AGR and 28 for SCR in the main setting reconcile with AGR's lower training time? Does Table 4 include anchor precomputation and all reward-serving GPU resources on an equal footing?
-
Equation (5) assigns rewards from observed wins, whereas Algorithm 1 assigns rewards from positions in a total order, including when conflict adjustment is disabled. Which is implemented, and how are ties handled? Also, please distinguish the optimal Kemeny objective in Property 3 from the greedy heuristic used in practice, which need not maximize weighted agreement.
6 comments
Response (2/6)
W2 & Q3: Novelty and Comparisons with Identical Judgments
Thanks for your helpful feedback! We would like to clarify our contribution and separate the existing strengthened PRC baselines from the additional shared-judgment experiment.
-
We build on established ranking methods. RRHF [1] aligns policy log-probabilities with preferences through a ranking loss. RRC converts a fixed judge's comparisons into scalar rewards for GRPO. Our contribution concerns how comparative judgments become policy-learning signals. We view reference comparisons and win-count aggregation as established techniques that support this construction.
-
Appendix C.2, Table 6 in the revised paper (Table 5 in the original submission) compares RRC with strengthened PRC baselines. On AE2, PRC with majority voting and soft aggregation score 32.8 and 30.1, compared with 36.4 for SCR and 35.8 for AGR. At the main group sizes, SCR uses 28 comparisons per vote and AGR uses 64. Please note that this table alone does not establish a shared query budget or identical opponents and judgment samples. To address your concern, the shared-judgment comparison below directly controls the comparison inputs.
-
Comparison with identical judgments. The additional offline comparison uses 100 prompts, eight responses and eight fixed anchors per prompt, and four judge samples per pair: 25,600 judgments in total. All methods reuse the same pairs, A/B assignments, reasoning traces, and token probabilities. The reasoning cap is 512 tokens.
Hard wins use a strict majority, with no win for a 2-2 split. Mean probabilities average judge probabilities before summing over anchors. Calibrated log-odds use Platt scaling fitted on 2,000 disjoint preference pairs, then average logits over samples and sum over anchors. Probabilities are normalized over the two preference labels and clipped to for logit computation.
Reward Construction Agreement with Model-Generated Ordering (%) Zero-Variance Groups (%) AE2 after Short RL Hard-win aggregation 84.68 0 32.7 Mean preference probabilities 91.11 0 31.9 Calibrated log-odds 92.43 0 33.1 In both offline comparisons (here and in Q6), the reference orderings are generated by the same model being trained. Agreement measures consistency with this model-generated reference, rather than accuracy against independent human annotations. We give half credit for tied predictions. Zero variance is measured before adding the format reward. The short RL runs use 50 optimizer steps with the same initial policy, anchors, and judge budget. Policy responses can diverge after updates, so identical judgment samples apply to the offline comparison only.
The results show that:
-
Calibrated log-odds achieves the highest offline agreement and short-run AE2 in this comparison. Hard wins score 0.8 points above mean probabilities on AE2, but 0.4 points below calibrated log-odds. These results do not establish that hard wins are uniformly better than probability-based rewards.
-
The log-odds result does not isolate a benefit from calibration. Without clipping saturation, a shared positive Platt transform preserves the aggregated ranking. It also cancels under ideal group normalization of the ranking reward alone. Format rewards, normalization epsilon, clipping, and reward scaling can affect the RL update.
-
None of the 100 groups has identical rewards under any of the three methods. This describes the evaluated sample, not the population rate. The AGR results in Q2 use a different sample and voting setting.
-
We promise to add these experiments and this discussion in the revision.
Reference
[1] Zheng Yuan et al. RRHF: Rank Responses to Align Language Models with Human Feedback without Tears. NeurIPS 2023.
Response (3/6)
Q2: Reward Ties and Anchor Coverage
Thanks for this important question! We would like to clarify that reference-policy anchors keep the comparison distribution fixed. Their quality coverage may still become limited as training progresses.
-
Anchor coverage. The 3B-scale, w/ Thinking comparison uses eight anchors without additional voting.
Anchor Distribution AE2 Zero-Variance Groups (%) All-Win Responses (%) Reference-policy anchors 35.8 8 22 Narrow-quality anchors 33.4 19 36 Anchors spanning quality levels 36.3 5 13 Zero-variance groups have identical ranking rewards before the format reward. All-win responses defeat every anchor.
- Broader anchor coverage reduces reward ties from 8% to 5% and all-win responses from 22% to 13%. AE2 increases from 35.8 to 36.3.
- Tied groups provide no ranking-based advantage, although format rewards may still differ.
-
Changes during training. We evaluate equal-size rollout samples on common prompts at 20%, 50%, and 90% of training. The refreshed strategy generates eight new anchors per prompt from the policy checkpoints at 40% and 70%.
Anchor Strategy Zero-Variance Groups: Early / Middle / Late (%) All-Win Responses: Early / Middle / Late (%) Final AE2 Fixed reference anchors 5 / 7 / 12 12 / 20 / 34 35.8 Fixed quality-diverse anchors 4 / 6 / 5 8 / 12 / 19 36.3 Periodically refreshed anchors 5 / 4 / 6 12 / 10 / 16 36.1 -
Fixed reference anchors lose discriminative power in this comparison. Their tie rate rises from 5% to 12%, and the all-win rate rises from 12% to 34%.
-
Both diverse and refreshed anchors reduce late saturation. Refreshing achieves AE2 of 36.1, compared with 35.8 for fixed reference anchors and 36.3 for fixed diverse anchors. Refreshing helps in this run, but does not outperform fixed diverse anchors.
The fixed-anchor aggregate rates above equal the averages over these three equally sized evaluation windows. They describe anchor coverage across training, rather than the second-half optimization diagnostics.
-
We promise to add these experiments in the revision.
Response (1/6)
Dear Reviewer yKNk,
We sincerely appreciate your constructive and thoughtful feedback.
We greatly appreciate your recognition that our paper is "clear and well organized" and that the distinction between preference-token confidence and response quality is "useful". We also appreciate your encouraging assessment of the empirical improvements and your acknowledgment of the additional controls in the appendix.
Below, we address the related weaknesses and questions together.
W1 & Q1: Margin Awareness and Ordinal Information
Thanks for your insightful comment! We would like to clarify the distinction between ordinal information and quality margins.
Appendix A gives a sufficient condition for preserving normalized advantages. It does not prove that win-count rewards satisfy this condition. In your example, qualities (0,1,100) yield SCR rewards (0,α,2α). The order is preserved, but the quality gaps are not. Binary rankings cannot distinguish quality scales related by a strictly increasing transformation.
For AGR, assume noiseless comparisons and independent anchor qualities drawn from a continuous CDF FA. For a fixed prompt,
E[ri−rj∣qi,qj]=αn(FA(qi)−FA(qj)).
Expected reward gaps measure percentile differences within the anchor distribution. They are proportional to quality gaps when FA is affine with positive slope over the relevant interval. Nearly constant positive density gives a local approximation. Finite anchors can still produce ties.
RRC can provide useful ordinal learning signals without recovering oracle margins.
We promise to revise this claim and add this discussion in the revision.