acceptodds
Under review as a conference paper at ICLR 2027

Flexible Quantile Reward Modeling for RLHF with Graded-Score Supervision

Abstract

Reward models are central to RLHF because they determine which model behaviors are reinforced during policy optimization. However, preference datasets with graded response scores are often reduced to binary chosen/rejected labels, discarding information about preference strength and reward location, while existing distributional approaches may impose restrictive parametric assumptions. We introduce the Flexible Quantile Reward Model (FQRM), which represents each response with a scalar-compatible center and an ordered quantile support capable of modeling skewed, asymmetric, and heavy-tailed reward distributions. Given comparable response-level scores, FQRM derives complementary supervision for relative ranking, preference strength, support location, and coarse reward regions, allowing the available graded information to constrain a single response-level representation. Unlike attribute-based quantile reward models, FQRM requires neither a predefined reward-attribute ontology nor attribute-level annotations. Experiments across multiple reward model benchmarks demonstrate the effectiveness of the proposed framework.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Reward Modeling, Quantile Reward Models, Preference Learning, RLHF

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_sdL1OpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

Flexible Quantile Reward Modeling (FQRM) represents each prompt–response pair with a strictly ordered empirical reward support whose mean serves as the scalar center used for ranking. It uses graded scores and preference strengths to train both the support and pairwise comparisons, without assuming a fixed reward distribution or requiring attribute-level annotations. The paper gives representation-level results for the support and reports improved ranking accuracy over scalar and Gaussian reward models across three benchmarks.

Critical strengths
  • The ordered support allows flexible reward shapes while retaining its mean as a scalar score for standard ranking.
  • The training objective uses graded scores and preference strengths to supervise both support locations and pairwise comparisons, without requiring attribute-level annotations.
  • Across three reward-model benchmarks, FQRM improves on the evaluated scalar and Gaussian baselines. Repeated runs and a support-size ablation provide further evidence that FQRM is effective.
Critical weaknesses
  • Proposition 1 in Section 4 is asymptotic in NN, whereas the method is evaluated mainly at N=32N=32. The paper gives neither finite-NN error bounds nor empirical support-size scaling evidence showing that N=32N=32 is sufficient.
  • The N=1N=1 versus N=32N=32 ablation changes both support capacity and how the training losses operate. Without fixed-NN loss ablations, the gains cannot be attributed specifically to the learned reward distribution.
  • The paper provides no repeated independent ratings for the same prompt–response pair. It therefore does not establish whether the learned support corresponds to the conditional distribution, dispersion, or disagreement of human judgments for that response.
  • The N2N^2 cross-support loss introduces additional training computation and potentially activation memory, but the authors report no wall-clock or peak-memory comparison across support sizes.
  • The Appendix D includes a PPO pilot, but it does not compare FQRM with a matched scalar reward model. The ranking gains therefore have not been shown to improve downstream policy optimization.
Questions for authors
  • With N=32N=32 fixed, what happens when the distributional win, pinball, or region-anchor loss is removed separately?
  • What are the training-time and peak-memory costs relative to a scalar reward model at different support sizes?
  • Can you compare downstream PPO using FQRM's scalar center against a matched scalar reward model?
  • How is the ambiguity-enriched training subset selected, and what are its source and category proportions?
7 comments
AuthorsOpenReview ↗

Response to Reviewer sdL1

Thank you for your specific suggestions. We added a finite-support error bound, loss ablations at fixed support size, training-cost measurements, and a downstream comparison under common settings. We also clarify what the learned support represents and how the training subset was constructed.

1. Finite-support approximation and the choice of N=32N=32

Let a target distribution PP be supported on [a,b][a,b], with D=b−aD=b-a and generalized quantile function QQ. At midpoint levels ui=i−12Nu_i=\frac{i-\frac{1}{2}}{N}, the equally weighted distribution PN∗P_N^{*} supported at Q(ui)Q(u_i) satisfies

W1(P,PN∗)≤D2N.W_1(P,P_N^*)\leq \frac{D}{2N}.

Within each quantile interval, monotonicity bounds the integrated midpoint-approximation error by the quantile variation divided by 2N2N. These variations sum to at most DD. The argument allows discrete ratings and repeated quantiles. For the learned support points, the triangle inequality further gives

W1(P,P^N)≤D2N+1N∑i∣q^i−Q(ui)∣.W_1(P,\widehat P_N)\leq \frac{D}{2N}+\frac{1}{N}\sum_i\left\lvert\widehat q_i-Q(u_i)\right\rvert.

This separates finite-resolution error from representation and learning error. On the normalized rating range [−2,2][-2,2], the first term is 0.0625 at N=32N=32.

We complemented this result with three runs each at N=16N=16 and 64, alongside the three existing runs each at N=1N=1 and 32. All models use fixed step-2,000 checkpoints. Values are means ±\pm sample standard deviations (%).

NNRewardBench-v1RewardBench-v2RM-Bench
170.68±22.7070.68\pm 22.7047.26±24.6547.26\pm 24.6562.22±12.6562.22\pm 12.65
1683.87±0.9383.87\pm 0.9360.66±1.1360.66\pm 1.1369.08±0.9569.08\pm 0.95
3284.55±0.5784.55\pm 0.5760.88±1.1160.88\pm 1.1168.81±0.1568.81\pm 0.15
6483.83±1.9183.83\pm 1.9161.24±0.4861.24\pm 0.4868.97±2.0168.97\pm 2.01

Increasing NN to 64 does not consistently improve performance. This supports N=32N=32 as a practical default.

2. What the learned support represents

FQRM models a predictive reward distribution using source-normalized response ratings as its feedback target. Graded ratings, score-derived preference strength, and pairwise rankings jointly supervise the ordered support points, while their center remains a scalar reward.

This target differs from identifying disagreement among human raters for a particular response. A single rating can contribute to quantile learning through model parameters shared across examples, but cannot by itself identify that response's human-rating distribution. Likewise, support width is not an estimate of epistemic uncertainty from a parameter posterior or model ensemble.

We tested the representation's predictive value against real human ratings in HelpSteer3, with model weights frozen and calibration fitted only on the training split. The results below show that retaining pinball improves prediction of rating distributions and low-rating probabilities. This provides evidence for a predictive interpretation of the support.

3. Individual loss contributions at fixed N=32N=32

Pinball follows established quantile regression, including its use in distributional reinforcement learning and QRM. Distributional-win follows the probabilistic BT construction used by PURM, averaging comparisons over reward distributions. Our method combines these objectives with ordered response-level supports to jointly use ratings, preference strength, and ranking signals.

ModelRB-v1RB-v2RM-Bench
Full FQRM84.55±0.5784.55\pm 0.5760.88±1.1160.88\pm 1.1168.81±0.1568.81\pm 0.15
Without distributional-win84.87±0.2284.87\pm 0.2260.54±0.6860.54\pm 0.6869.02±0.6869.02\pm 0.68
Without pinball85.27±1.2285.27\pm 1.2263.42±2.2163.42\pm 2.2170.32±0.8070.32\pm 0.80
Without region anchor73.47±19.1273.47\pm 19.1246.31±24.8346.31\pm 24.8361.83±13.2261.83\pm 13.22

Removing the anchor lowers average performance and increases variation across seeds, consistent with its location constraint helping stabilize some runs.

Pinball and distributional-win also supervise predictive targets that center-ranking accuracy alone cannot measure. We tested these targets on real HelpSteer3 human feedback.CRPS evaluates the predictive rating distribution against observed human ratings; the Brier score measures squared error in predicted event probabilities, here for low ratings and pairwise preferences. Both are established scoring rules in probabilistic prediction, reported here in loss form so that lower is better.

ModelRating-distribution CRPS ↓Low-rating probability Brier ↓Preference probability Brier ↓
Full FQRM0.7321±0.01710.7321\pm 0.01710.1498±0.00250.1498\pm 0.00250.2063±0.00170.2063\pm 0.0017
Without pinball0.9048±0.00130.9048\pm 0.00130.1660±0.00230.1660\pm 0.00230.2096±0.00430.2096\pm 0.0043
Without distributional-win0.7164±0.01360.7164\pm 0.01360.1466±0.00400.1466\pm 0.00400.2128±0.00280.2128\pm 0.0028

Pinball supervises the rating distribution of each response. Retaining it reduces CRPS by 19.1% and Brier error for low-helpfulness ratings (levels 0 or 1) by 9.8%. Retaining it gives more accurate predictions of rating distributions and low-rating probabilities.

Distributional-win supervises preference probabilities between responses. Retaining it reduces preference Brier from 0.2128 to 0.2063. Comparing all support points therefore provides information about preference probabilities beyond which scalar center is larger.

HelpSteer3 is a published human-feedback dataset covering diverse tasks and languages. We further compared model performance on the dataset. The table below reports mean agreement with human preferences:

Full FQRMWithout pinballWithout distributional-win
71.96±1.1571.96\pm 1.1570.71±2.2970.71\pm 2.2971.33±0.4771.33\pm 0.47

4. Training time and memory as NN increases

To quantify the training overhead of increasing the support size, we timed 30 CUDA-synchronized optimizer updates after 50 warmup updates. The setup uses two A800 80GB GPUs, input length 1,024, two response pairs per microbatch per GPU, four gradient-accumulation steps, an effective batch of 16 pairs, BF16, and gradient checkpointing. All configurations use the same pair and token workloads; memory is the larger peak allocation across the two GPUs.

NNSeconds/updatePeak allocated memory/GPU (GiB)
13.77818.044
163.77518.045
323.77118.045
643.78718.046

The small overhead is consistent with the implementation: all support points for a response share a representation from one Transformer computation. The quadratic operation compares support values, rather than repeating the Transformer forward pass N2N^{2} times. These measurements show a small overhead for the tested 7B backbone and N≤64N\leq 64.

5. Downstream comparison under common settings

We added a GRPO comparison under common settings, reported separately from the historical PPO pilot. Scalar BT, PURM, and FQRM start from the same initial policy and use identical prompts, prompt order, and settings for 1,024 updates, with policy seed 42. FQRM uses its scalar center as the reward.

Evaluation uses 256 held-out prompts, a 512-token response budget, and three DeepSeek Flash judgments per answer pair. Wins, ties, and losses receive 1, 0.5, and 0 points, respectively. Confidence intervals are obtained by resampling prompts.

ComparisonFQRM preference rate95% interval
FQRM vs BT54.88%[50.26%,59.25%]
FQRM vs PURM55.60%[51.11%,60.03%]

The FQRM-trained policy receives higher preference scores than both baselines, providing evidence of its downstream benefits under the same optimization settings.

6. Construction and composition of the ambiguity-enriched subset

To clarify the composition of the training subset, we summarize the sample counts and proportions for each source below. The training data contain 20,495 response pairs from 17,763 distinct prompts:

SourceTraining response pairsShare
UltraFeedback-binarized9,80047.82%
preference_700K9,80047.82%
HelpSteer28954.37%

We construct pairs from existing ratings, excluding equal-score pairs and assigning the higher-scoring response as chosen. Within each source, selection prioritizes small rating gaps while retaining clearly separated anchor pairs. For a soft preference pp derived from the rating gap, 1−∣2p−1∣1-\lvert 2p-1\rvert measures proximity to a tie, and ∣2p−1∣\lvert 2p-1\rvert measures preference clarity. This enriches ambiguity in the rating-gap sense; it does not claim to measure disagreement among raters. Anchor pairs used in data selection are distinct from the region-anchor training loss.

After sample selection, each source's median and IQR are fitted only to chosen/rejected ratings in the training set. Normalization uses

\frac{\text{score} - \text{median}}{\max(\text{IQR}, 10^{-6})}, -2, 2 \right)$$ with the same statistics applied to held-out data. The final preference target is $$p = \operatorname{clip}\left( \sigma(z_{\text{chosen}} - z_{\text{rejected}}), 0.55, 0.98 \right)$$ and the region boundaries are $-0.5$ and $+0.5$. HelpSteer2's five existing attribute ratings are combined into a scalar during preprocessing; FQRM requires no additional attribute annotations or separate attribute reward heads. Sample selection is not stratified by semantic task category, and the stored fields do not support reliable category proportions. The table therefore reports the actual source proportions without treating sources as task categories.
Reviewer_3VdbOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper proposes the Flexible Quantile Reward Model (FQRM), which extends conventional scalar reward modeling by representing reward distributions through ordered quantile supports rather than point estimates or fixed Gaussian distributions. FQRM further leverages graded response scores to jointly supervise reward location, preference strength, and pairwise ranking, without requiring predefined reward attributes. Empirical results across three reward-model benchmarks show that FQRM outperforms scalar and Gaussian probabilistic baselines, particularly in preference ranking tasks.

Critical strengths
  • FQRM replaces fixed Gaussian assumptions with ordered quantile supports, allowing the model to represent more complex reward distributions, including skewed and asymmetric shapes.
  • FQRM combines response-level scores and score differences into a unified training objective, providing information about reward values and preference strength beyond traditional binary preference labels.
Critical weaknesses
  • Although FQRM achieves strong results on reward-model benchmarks, the downstream PPO experiments only evaluate variants of FQRM, without comparing against other reward models under the same conditions. The support-size ablation only considers N = 1 and N = 32, and the contributions of individual loss components are not examined. Moreover, several baselines show large variations across random seeds without sufficient explanation, while important training details are not clearly reported. These limitations make it difficult to determine which components drive the improvements, whether the comparisons are reliable, and whether FQRM provides meaningful benefits for downstream RLHF.
  • FQRM introduces multiple quantile supports and several training objectives, making the method relatively complex. However, the paper does not sufficiently explain why such a flexible distributional representation is necessary. Although the N = 1 ablation shows that multiple support points improve performance, it does not distinguish the benefit of modeling reward variation from that of learning flexible distribution shapes. Since the final ranking relies only on the scalar center, it remains unclear whether a simpler model, such as one with a learned center and scale but fixed quantile spacing, could achieve comparable performance. This raises concerns about whether the additional complexity is justified.
Questions for authors
  • Can the authors provide a comparison of policies trained with FQRM and other reward models (e.g., Scalar BT or PURM) under the same PPO or GRPO settings? This would help clarify whether the improvements on reward-model benchmarks translate into better downstream RLHF performance.
  • Can the authors clarify the individual contributions of the loss components and whether all of them are necessary? Additionally, how would FQRM compare with a simpler model that uses fixed quantile spacing with a learned center and scale?
  • What explains the large performance variance of several baseline methods across random seeds? Could the authors provide more details on the training configurations and optimization settings to help understand these differences?
3 comments
AuthorsOpenReview ↗

Response to Reviewer 3Vdb

Thank you for your suggestions. We added downstream comparisons under the same settings, support-size ablations, loss ablations at fixed NN, and the proposed location–scale control. We also evaluated the existing models using human ratings and preference data to clarify the contributions of the distributional objectives beyond scalar ranking.

Q1. Downstream comparison with BT and PURM

We trained policies with Scalar BT, PURM, and FQRM under the same GRPO configuration, using the same initial policy, training prompts, and prompt order, with 1,024 updates for each. Evaluation used the same 256 held-out prompts, with DeepSeek Flash making three judgments per answer pair.

Policy comparisonFQRM preference score95% confidence interval
FQRM vs BT54.88%[50.26%, 59.25%][50.26\%,\,59.25\%]
FQRM vs PURM55.60%[51.11%, 60.03%][51.11\%,\,60.03\%]

The FQRM-trained policy receives higher preference scores than both baselines, providing evidence of its downstream benefits under the same optimization settings.

Q2. Support size, loss components, and the location–scale control

Support size. To examine how support size affects the results, we extended the ablation to N=16N=16 and N=64N=64:

NNRB-v1RB-v2RM-Bench
170.68±22.7070.68\pm 22.7047.26±24.6547.26\pm 24.6562.22±12.6562.22\pm 12.65
1683.87±0.9383.87\pm 0.9360.66±1.1360.66\pm 1.1369.08±0.9569.08\pm 0.95
3284.55±0.5784.55\pm 0.5760.88±1.1160.88\pm 1.1168.81±0.1568.81\pm 0.15
6483.83±1.9183.83\pm 1.9161.24±0.4861.24\pm 0.4868.97±2.0168.97\pm 2.01

Increasing NN to 64 does not bring consistent improvements, supporting N=32N=32 as a practical setting. Since N=1N=1 also changes how the distributional losses operate, we separately ablated the losses at fixed N=32N=32.

Loss components. Pinball follows established quantile regression, including its use in distributional reinforcement learning and QRM. Distributional-win follows the probabilistic BT construction used by PURM, averaging comparisons over reward distributions. Our method combines these objectives with ordered response-level supports to jointly use ratings, preference strength, and ranking signals.

ModelRB-v1RB-v2RM-Bench
Full FQRM84.55±0.5784.55\pm 0.5760.88±1.1160.88\pm 1.1168.81±0.1568.81\pm 0.15
Without distributional-win84.87±0.2284.87\pm 0.2260.54±0.6860.54\pm 0.6869.02±0.6869.02\pm 0.68
Without pinball85.27±1.2285.27\pm 1.2263.42±2.2163.42\pm 2.2170.32±0.8070.32\pm 0.80
Without region anchor73.47±19.1273.47\pm 19.1246.31±24.8346.31\pm 24.8361.83±13.2261.83\pm 13.22

Removing the anchor lowers average performance and increases variation across seeds, consistent with its location constraint helping stabilize some runs.

Pinball and distributional-win also supervise predictive targets that center-ranking accuracy alone cannot measure. We tested these targets on real HelpSteer3 human feedback.CRPS evaluates the predictive rating distribution against observed human ratings; the Brier score measures squared error in predicted event probabilities, here for low ratings and pairwise preferences. Both are established scoring rules in probabilistic prediction, reported here in loss form so that lower is better.

ModelRating-distribution CRPS ↓Low-rating probability Brier ↓Preference probability Brier ↓
Full FQRM0.7321±0.01710.7321\pm 0.01710.1498±0.00250.1498\pm 0.00250.2063±0.00170.2063\pm 0.0017
Without pinball0.9048±0.00130.9048\pm 0.00130.1660±0.00230.1660\pm 0.00230.2096±0.00430.2096\pm 0.0043
Without distributional-win0.7164±0.01360.7164\pm 0.01360.1466±0.00400.1466\pm 0.00400.2128±0.00280.2128\pm 0.0028

Pinball supervises the rating distribution of each response. Retaining it reduces CRPS by 19.1% and Brier error for low-helpfulness ratings (levels 0 or 1) by 9.8%. Retaining it gives more accurate predictions of rating distributions and low-rating probabilities.

Distributional-win supervises preference probabilities between responses. Retaining it reduces preference Brier from 0.2128 to 0.2063. Comparing all support points therefore provides information about preference probabilities beyond which scalar center is larger.

HelpSteer3 is a published human-feedback dataset covering diverse tasks and languages. We further compared model performance on the dataset. The table below reports mean agreement with human preferences:

Full FQRMWithout pinballWithout distributional-win
71.96±1.1571.96\pm 1.1570.71±2.2970.71\pm 2.2971.33±0.4771.33\pm 0.47
AuthorsOpenReview ↗

Response to Reviewer 3Vdb

Location–scale control. Following your suggestion, we fixed the relative support spacing and learned only location and scale, retaining N=32N=32 and using the same training objectives and loss weights:

ParameterizationRB-v1RB-v2RM-Bench
Fixed spacing, learned location and scale84.00±1.2084.00\pm 1.2060.01±1.7660.01\pm 1.7669.13±0.3269.13\pm 0.32
Full FQRM84.55±0.5784.55\pm 0.5760.88±1.1160.88\pm 1.1168.81±0.1568.81\pm 0.15

The overall results on these three benchmarks are close. JudgeBench (Tan et al., ICLR 2025) is a challenging paired-response benchmark for LLM judges and reward models, covering knowledge, reasoning, mathematics, and coding; it requires distinguishing objectively correct responses from responses containing subtle errors. We further compared the two models on JudgeBench and HelpSteer3 preference prediction:

EvaluationFQRMLocation–scale
JudgeBench, 620 pairs, accuracy (%) ↑61.88±1.9261.88\pm 1.9260.00±1.1360.00\pm 1.13
HS3 preference sample, 500 pairs, Brier ↓0.1975±0.00170.1975\pm 0.00170.1997±0.00320.1997\pm 0.0032
Multilingual HS3 sample, 204 pairs, Brier ↓0.1656±0.00190.1656\pm 0.00190.1703±0.00320.1703\pm 0.0032

FQRM achieves 1.88 percentage points higher mean JudgeBench accuracy and lower mean preference-probability errors on both HS3 samples.

Q3. Training configuration and variation across seeds

In response to your questions about training details and variation across seeds, we reviewed the training configurations. The table below summarizes the shared settings used in the new reward-model experiments:

SettingConfiguration
Backbone and dataQwen2.5-7B-Instruct; 20,495 preference pairs from three sources; source-normalized ratings
Seeds and budget42, 123, 456; 2,000 updates; validation and saving every 200 updates
Effective batch and input length16 preference pairs per update; 1,024 tokens
Optimizer and scheduleAdamW, learning rate 2×10−42\times 10^{-4}, 50 warmup updates, WarmupDecayLR
Adam parameters and regularizationβ=(0.9, 0.999)\beta=(0.9,\,0.999), ε=10−8\varepsilon=10^{-8}, weight decay 0.01, gradient clipping 1.0
LoRA and memory settingsRank 16, alpha 32, dropout 0.05; attention and MLP projections; BF16, gradient checkpointing, ZeRO-2

Thank you again for your detailed review comments. We appreciate your feedback on whether our responses have addressed all your concerns, and would be happy to provide further explanations if any questions remain.

Reviewer_RSVVOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

FQRM predicts ordered reward quantiles for each response and uses their mean as a standard reward score. It combines response ratings, preference strength inferred from rating gaps, and pairwise rankings, without attribute-level labels or a Gaussian assumption. It outperforms the scalar BT and PURM on three reward-model benchmarks.

Critical strengths

FQRM learns a non-Gaussian quantile representation from overall ratings and pairwise preferences while retaining a scalar reward. Strong performance on challenging categories and under surface perturbations.

Critical weaknesses
  1. The baseline scores are unusually low: Scalar BT reaches only 50.8% on RewardBench-v1 and PURM 62.0%, while BT reward models with similarly sized backbones often reach 75–85%. The 2,000-step LoRA budget may be insufficient, and the paper does not identify the backbone. 2.On the ambiguity enriched data, scalar BT does not use preference-strength information. The paper should also compare against strength-aware methods, such as adaptive-margin training from Secrets of RLHF in Large Language Models Part II: Reward Modeling.
Questions for authors
  1. Were the baselines trained sufficiently? Which backbone was used, and did Scalar BT and PURM converge or receive method-specific tuning within the 2,000-step budget?
  2. Given the stronger results of generative reward models, is further work on BT-style models still worthwhile?
2 comments
AuthorsOpenReview ↗

Response to Reviewer RSVV

Thank you for your suggestions on baseline training and the practical value of FQRM. We examined the archived training trajectories, added a preference-strength-aware BT baseline over three seeds, and measured the inference latency of FQRM and a generative reward model on identical inputs.

Q1. Baseline training and preference-strength-aware comparisons

Regarding your questions about the backbone and training budget, all methods use Qwen2.5-7B-Instruct, the same three-source training mixture, and a budget of 2,000 updates. The shared main settings are an effective batch size of 16 preference pairs, a learning rate of 2×10−42\times 10^{-4}, 50 warmup updates, weight decay of 0.01, and LoRA rank/alpha/dropout of 16/32/0.05. These comparisons use a fixed training budget, without method-specific hyperparameter searches.

To address the low scores at the end of training, we examined intermediate checkpoints. The highest observed RewardBench-v1 scores in the archived source-normalized seed-42 trajectories are:

MethodStep at historical peakRewardBench-v1 (%)
Scalar BT40074.24
PURM60068.68
FQRM1,60085.29

The baselines achieved substantially higher scores earlier in training than at the endpoint. For example, in this BT run, the score declined from 74.24% at step 400 to 51.40% at step 2,000. FQRM learned effective rankings early in its archived trajectory and broadly maintained this performance through subsequent training.

Preference-strength-aware BT. Following your recommendation of Secrets of RLHF in Large Language Models Part II: Reward Modeling, we added an adaptive-margin BT control. It uses the same three-source mixture and source-normalized ratings as FQRM, with the nonnegative rating gap as the margin and a coefficient of 1. This applies the preference-strength-aware approach in the cited work to our existing rating information.

MethodRewardBench-v1RB-v2RM-Bench
Adaptive-margin BT61.52±17.0361.52\pm 17.0332.18±23.1832.18\pm 23.1854.54±12.4954.54\pm 12.49
FQRM, N=32N=3284.55±0.5784.55\pm 0.5760.88±1.1160.88\pm 1.1168.81±0.1568.81\pm 0.15

With this data and training budget, FQRM achieves higher mean scores on all three benchmarks even when the scalar baseline also receives preference-strength information.

Q2. Practical value alongside generative reward models

Direct scoring remains useful when reward evaluations are repeated frequently, as in candidate selection and policy optimization. Recent research continues to explore this direction: Structural Reward Model (EMNLP 2025) identifies sequential decoding as an obstacle to efficient deployment; You Only Judge Once (2026) proposes a discriminative multimodal reward model that scores in a single forward pass and demonstrates its use in GRPO. RM-R1, Appendix J also discusses the inference cost of long reasoning chains.

FQRM learns a richer reward representation while retaining an interface that produces a scalar score from one backbone forward pass per response. As a literature reference, the published results of the similarly sized RM-R1 model are close to ours:

Method and result sourceRewardBench-v1RM-Bench
FQRM84.55±0.5784.55\pm 0.5768.81±0.1568.81\pm 0.15
RM-R1-Qwen2.5-Instruct-7B85.270.2

We also timed the two existing 7B models on the same A800 80GB GPU. Both used the same 10 RewardBench-v1 questions, each containing two complete candidate responses; after model loading, two additional questions were used for warmup. FQRM scored both responses in one batch, while RM-R1 generated judgments in its native format.

Scoring procedureMean time per questionTotal for ten questionsMeasured GPU-hours
FQRM, one batched forward pass0.077 s0.771 s0.000214
RM-R1, one generated judgment12.304 s123.038 s0.034177
RM-R1, both response orders combined24.319 s243.193 s0.067554

In this measurement, FQRM required approximately 1160\frac{1}{160} of the time for one RM-R1 judgment. This demonstrates the practical value of retaining direct scoring while achieving competitive reported reward-model scores.

Thank you again for your detailed review comments. We appreciate your feedback on whether our responses have addressed all your concerns, and would be happy to provide further explanations if any questions remain.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.