- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
Flexible Quantile Reward Modeling (FQRM) represents each prompt–response pair with a strictly ordered empirical reward support whose mean serves as the scalar center used for ranking. It uses graded scores and preference strengths to train both the support and pairwise comparisons, without assuming a fixed reward distribution or requiring attribute-level annotations. The paper gives representation-level results for the support and reports improved ranking accuracy over scalar and Gaussian reward models across three benchmarks.
Critical strengths
- The ordered support allows flexible reward shapes while retaining its mean as a scalar score for standard ranking.
- The training objective uses graded scores and preference strengths to supervise both support locations and pairwise comparisons, without requiring attribute-level annotations.
- Across three reward-model benchmarks, FQRM improves on the evaluated scalar and Gaussian baselines. Repeated runs and a support-size ablation provide further evidence that FQRM is effective.
Critical weaknesses
- Proposition 1 in Section 4 is asymptotic in , whereas the method is evaluated mainly at . The paper gives neither finite- error bounds nor empirical support-size scaling evidence showing that is sufficient.
- The versus ablation changes both support capacity and how the training losses operate. Without fixed- loss ablations, the gains cannot be attributed specifically to the learned reward distribution.
- The paper provides no repeated independent ratings for the same prompt–response pair. It therefore does not establish whether the learned support corresponds to the conditional distribution, dispersion, or disagreement of human judgments for that response.
- The cross-support loss introduces additional training computation and potentially activation memory, but the authors report no wall-clock or peak-memory comparison across support sizes.
- The Appendix D includes a PPO pilot, but it does not compare FQRM with a matched scalar reward model. The ranking gains therefore have not been shown to improve downstream policy optimization.
Questions for authors
- With fixed, what happens when the distributional win, pinball, or region-anchor loss is removed separately?
- What are the training-time and peak-memory costs relative to a scalar reward model at different support sizes?
- Can you compare downstream PPO using FQRM's scalar center against a matched scalar reward model?
- How is the ambiguity-enriched training subset selected, and what are its source and category proportions?
7 comments
3. Individual loss contributions at fixed
Pinball follows established quantile regression, including its use in distributional reinforcement learning and QRM. Distributional-win follows the probabilistic BT construction used by PURM, averaging comparisons over reward distributions. Our method combines these objectives with ordered response-level supports to jointly use ratings, preference strength, and ranking signals.
| Model | RB-v1 | RB-v2 | RM-Bench |
|---|---|---|---|
| Full FQRM | |||
| Without distributional-win | |||
| Without pinball | |||
| Without region anchor |
Removing the anchor lowers average performance and increases variation across seeds, consistent with its location constraint helping stabilize some runs.
Pinball and distributional-win also supervise predictive targets that center-ranking accuracy alone cannot measure. We tested these targets on real HelpSteer3 human feedback.CRPS evaluates the predictive rating distribution against observed human ratings; the Brier score measures squared error in predicted event probabilities, here for low ratings and pairwise preferences. Both are established scoring rules in probabilistic prediction, reported here in loss form so that lower is better.
| Model | Rating-distribution CRPS ↓ | Low-rating probability Brier ↓ | Preference probability Brier ↓ |
|---|---|---|---|
| Full FQRM | |||
| Without pinball | |||
| Without distributional-win |
Pinball supervises the rating distribution of each response. Retaining it reduces CRPS by 19.1% and Brier error for low-helpfulness ratings (levels 0 or 1) by 9.8%. Retaining it gives more accurate predictions of rating distributions and low-rating probabilities.
Distributional-win supervises preference probabilities between responses. Retaining it reduces preference Brier from 0.2128 to 0.2063. Comparing all support points therefore provides information about preference probabilities beyond which scalar center is larger.
HelpSteer3 is a published human-feedback dataset covering diverse tasks and languages. We further compared model performance on the dataset. The table below reports mean agreement with human preferences:
| Full FQRM | Without pinball | Without distributional-win |
|---|---|---|
4. Training time and memory as increases
To quantify the training overhead of increasing the support size, we timed 30 CUDA-synchronized optimizer updates after 50 warmup updates. The setup uses two A800 80GB GPUs, input length 1,024, two response pairs per microbatch per GPU, four gradient-accumulation steps, an effective batch of 16 pairs, BF16, and gradient checkpointing. All configurations use the same pair and token workloads; memory is the larger peak allocation across the two GPUs.
| Seconds/update | Peak allocated memory/GPU (GiB) | |
|---|---|---|
| 1 | 3.778 | 18.044 |
| 16 | 3.775 | 18.045 |
| 32 | 3.771 | 18.045 |
| 64 | 3.787 | 18.046 |
The small overhead is consistent with the implementation: all support points for a response share a representation from one Transformer computation. The quadratic operation compares support values, rather than repeating the Transformer forward pass times. These measurements show a small overhead for the tested 7B backbone and .
5. Downstream comparison under common settings
We added a GRPO comparison under common settings, reported separately from the historical PPO pilot. Scalar BT, PURM, and FQRM start from the same initial policy and use identical prompts, prompt order, and settings for 1,024 updates, with policy seed 42. FQRM uses its scalar center as the reward.
Evaluation uses 256 held-out prompts, a 512-token response budget, and three DeepSeek Flash judgments per answer pair. Wins, ties, and losses receive 1, 0.5, and 0 points, respectively. Confidence intervals are obtained by resampling prompts.
| Comparison | FQRM preference rate | 95% interval |
|---|---|---|
| FQRM vs BT | 54.88% | [50.26%,59.25%] |
| FQRM vs PURM | 55.60% | [51.11%,60.03%] |
The FQRM-trained policy receives higher preference scores than both baselines, providing evidence of its downstream benefits under the same optimization settings.
6. Construction and composition of the ambiguity-enriched subset
To clarify the composition of the training subset, we summarize the sample counts and proportions for each source below. The training data contain 20,495 response pairs from 17,763 distinct prompts:
| Source | Training response pairs | Share |
|---|---|---|
| UltraFeedback-binarized | 9,800 | 47.82% |
| preference_700K | 9,800 | 47.82% |
| HelpSteer2 | 895 | 4.37% |
We construct pairs from existing ratings, excluding equal-score pairs and assigning the higher-scoring response as chosen. Within each source, selection prioritizes small rating gaps while retaining clearly separated anchor pairs. For a soft preference derived from the rating gap, measures proximity to a tie, and measures preference clarity. This enriches ambiguity in the rating-gap sense; it does not claim to measure disagreement among raters. Anchor pairs used in data selection are distinct from the region-anchor training loss.
After sample selection, each source's median and IQR are fitted only to chosen/rejected ratings in the training set. Normalization uses
\frac{\text{score} - \text{median}}{\max(\text{IQR}, 10^{-6})}, -2, 2 \right)$$ with the same statistics applied to held-out data. The final preference target is $$p = \operatorname{clip}\left( \sigma(z_{\text{chosen}} - z_{\text{rejected}}), 0.55, 0.98 \right)$$ and the region boundaries are $-0.5$ and $+0.5$. HelpSteer2's five existing attribute ratings are combined into a scalar during preprocessing; FQRM requires no additional attribute annotations or separate attribute reward heads. Sample selection is not stratified by semantic task category, and the stored fields do not support reliable category proportions. The table therefore reports the actual source proportions without treating sources as task categories.
Response to Reviewer sdL1
Thank you for your specific suggestions. We added a finite-support error bound, loss ablations at fixed support size, training-cost measurements, and a downstream comparison under common settings. We also clarify what the learned support represents and how the training subset was constructed.
1. Finite-support approximation and the choice of N=32
Let a target distribution P be supported on [a,b], with D=b−a and generalized quantile function Q. At midpoint levels ui=Ni−21, the equally weighted distribution PN∗ supported at Q(ui) satisfies
W1(P,PN∗)≤2ND.
Within each quantile interval, monotonicity bounds the integrated midpoint-approximation error by the quantile variation divided by 2N. These variations sum to at most D. The argument allows discrete ratings and repeated quantiles. For the learned support points, the triangle inequality further gives
W1(P,PN)≤2ND+N1∑i∣qi−Q(ui)∣.
This separates finite-resolution error from representation and learning error. On the normalized rating range [−2,2], the first term is 0.0625 at N=32.
We complemented this result with three runs each at N=16 and 64, alongside the three existing runs each at N=1 and 32. All models use fixed step-2,000 checkpoints. Values are means ± sample standard deviations (%).
Increasing N to 64 does not consistently improve performance. This supports N=32 as a practical default.
2. What the learned support represents
FQRM models a predictive reward distribution using source-normalized response ratings as its feedback target. Graded ratings, score-derived preference strength, and pairwise rankings jointly supervise the ordered support points, while their center remains a scalar reward.
This target differs from identifying disagreement among human raters for a particular response. A single rating can contribute to quantile learning through model parameters shared across examples, but cannot by itself identify that response's human-rating distribution. Likewise, support width is not an estimate of epistemic uncertainty from a parameter posterior or model ensemble.
We tested the representation's predictive value against real human ratings in HelpSteer3, with model weights frozen and calibration fitted only on the training split. The results below show that retaining pinball improves prediction of rating distributions and low-rating probabilities. This provides evidence for a predictive interpretation of the support.