- Rating
- 3
- Confidence
- 3
- Informativeness
- 3
- Presentation
- 2
- Soundness
- 3
Read review
Paper summary
The paper identifies the lack of confidence information in scalar reward models as a central, under-examined limitation of RLHF: GRPO standardizes rewards within each prompt group under the implicit assumption that all rewards are equally informative, which can overweight noisy, biased, or spuriously inflated scores. UARM addresses this as follows: (i) the scalar reward head is replaced by jointly trained quantile heads optimized with pinball loss (Eq. 5), with the median serving as the point reward consumed by GRPO; (ii) adaptive prediction intervals are constructed by split-conformal calibration over the shortest contiguous block of nested interquantile intervals (Eqs. 7–9), which automatically widens in low-density regions; (iii) each interval half-width is converted into a per-sample observation noise variance under a local Gaussian approximation (Eq. 12), which is then subtracted from the group variance to recover the signal variance (Eq. 13); and (iv) GRPO advantages are reweighted by a sample-specific signal-to-noise attenuation factor (Eq. 14), which reduces exactly to vanilla GRPO when noise is uniform across the group. UARM is evaluated on three preference datasets (HelpSteer, UltraFeedback, PKU-SafeRLHF) against a dozen comparators spanning the model-based UQ line (MC-Dropout, DER, Deep/Packed Ensembles, MCNF) and the distribution-free line (SCP, CQR, WCP, ACI, SCCP, CLEAR), across 8B–70B backbones (Qwen3-8B, LLaMA3.1-8B/70B), and on four safety-oriented downstream benchmarks (HarmBench, FFT, DAN, WildGuardMix) with two policy models.
Critical strengths
- Tight alignment between claims, evidence, and ablations. Table 2 cleanly disentangles the two components: removing conformal calibration (UARM) raises MSE by 21.7% on UltraFeedback (0.383 → 0.466), while replacing dense quantiles with three quantiles (UARM) reduces from 0.794 to 0.747. The full model is best on every dataset and metric, supporting the claimed synergy rather than a single-component story.
- Theoretically honest coverage analysis. Finite-sample marginal coverage (Theorem 3.1, Assumptions B.1/B.3) and asymptotic conditional coverage (Theorem 3.2, Assumptions B.2–B.4) are kept clearly separate; the conditional result states its required uniform quantile calibration, its deficit rate , and its failure regimes — notably, Remark B.7 explicitly states that the result applies to a fresh split and does not cover substantial policy-induced shift without covariate-shift correction. Marginal coverage is not overstated as conditional.
- Comprehensive baselines, consistent gains, negligible overhead. The benchmark suite covers essentially the full spectrum of modern UQ, and UARM achieves state-of-the-art uncertainty-ranked prediction on all three datasets, with gains that hold consistently across 8B–70B reward-model backbones (Table 3). Because the intervals are read directly from the calibrated quantile heads, no additional forward passes are required at inference — a meaningful advantage over ensemble methods.
- Clear and reproducible presentation. The Figure 1 case study is an effective intuitive anchor, and the reproduction details in Appendix C (score computation, training schedules, judge prompts) are thorough.
Critical weaknesses
- Missing highly related work: CFA (Conformal Feedback Alignment, arXiv:2601.17329). CFA already grounds preference weighting in conformal prediction: it constructs conformal prediction sets with controllable coverage to quantify answer-level reliability, aggregates set memberships into a per-pair weight via a set-wise aggregation rule, and injects that weight into both PPO-style reward-model training and the DPO preference loss. Given that Section A.3 explicitly positions UARM against uncertainty-aware policy-optimization work (ULPS, UPO, TUR-DPO), omitting the closest conformal-prediction-based alignment method is a substantive gap rather than a routine missing citation, and the axes of differentiation are currently left entirely to the reader.
- The "improves RLHF / code alignment" claim is established only on safety. All four downstream benchmarks measure jailbreak robustness or harmlessness; there is no helpfulness, instruction-following, or general-capability evaluation (e.g., AlpacaEval, MT-Bench, Arena-Hard, or knowledge/reasoning suites). Because the method downweights high-uncertainty rewards, and in a safety-oriented pipeline such rewards may coincide with responses that merely rephrase refusals, the current results cannot distinguish genuine alignment gains from a "safety by evasion" artifact, nor can they rule out degraded usefulness. The qualitative case study (Tables 5–6) shows the same pattern — refusals versus compliance — and does not close this gap.
- The @50 protocol is not comparable across methods, and full-set metrics are absent. Each uncertainty-aware method selects its own top-50% most uncertain samples (the Naive baseline instead uses a random 50% subset), so /MSE@50 are computed on different subsets for every method; a method can look better partly by defining a subset on which it is accurate. Full-test-set metrics, metrics on a common fixed subset, and risk–coverage curves are needed for a fair comparison. In addition, the PKU-SafeRLHF result — vs. 0.850 (Naive) and 0.940 (CLEAR), with MSE shrinking from 0.173 to 0.013, a roughly 13 reduction — is a far larger jump than anything observed on HelpSteer or UltraFeedback, and possible causes (the discrete 0–3 severity labels interacting with interval construction, subset-selection effects, or calibration/train leakage) should be explicitly ruled out.
- The core assumptions of Eq. (14) are not empirically tested. The conversion of interval half-width to a Gaussian noise variance (Eq. 12) is a local Gaussian approximation, and the conformal step relies on exchangeability between calibration points and online rollouts. Under GRPO the policy — and hence the reward distribution — drifts over update steps; the paper acknowledges this only as future work (Section 5) and provides no coverage tracking across the 20 updates, no empirical check of the Gaussian approximation (e.g., residual or PIT diagnostics), and no comparison with online recalibration.
- Limited downstream experimental rigor. The 20-update budget is very short, Table 4 reports no multi-seed variance or error bars, and downstream fine-tuning uses only two 8B policy models despite the 8B–70B scaling evidence for the reward models. Effect sizes as large as the 35.09% relative improvement on FFT (LLaMA3.1-8B) cannot be assessed for stability without repeated runs.
Questions for authors
- Relation to CFA. Please state the exact differences along at least the following axes: (a) the object whose uncertainty is modeled — reward-model uncertainty over the scalar reward vs. answer-level reliability of the two responses in a preference pair; (b) the conformal construction — regression intervals from quantile heads via shortest contiguous nested blocks vs. membership of responses in two sets at different coverage levels, defined by NLL-based (white-box) or frequency/entropy/similarity-based (black-box) nonconformity scores; (c) where the weight enters the loss — an SNR attenuation factor on per-rollout GRPO advantages during online policy optimization, including the signal/noise variance decomposition of Eqs. 13–14, vs. a per-pair scalar weight multiplying the RM preference loss (PPO-style) or the DPO loss; and (d) granularity — per-rollout reward instances vs. per-preference-pair weights. Are the two approaches complementary, and could they be combined?
- Beyond safety. Please add (or commit to adding) helpfulness/general-capability evaluations alongside the safety scores, longer GRPO training beyond 20 updates with reward-over-optimization curves, and multi-seed runs with standard deviations for Table 4.
- Fair head-to-head metrics. Please provide full-test-set metrics (or metrics on one common fixed subset) for Table 1, risk–coverage and width–error curves, and a concrete explanation of the anomalously large PKU-SafeRLHF gap, including how the discrete severity labels and the method-specific @50 subset selection contribute to it.
- Validity under policy drift. Can the authors report empirical coverage stratified by interval width, reward magnitude, or training step over the course of GRPO updates, together with PIT/QQ-style checks of the Gaussian-noise interpretation in Eq. (12)? If exchangeability degrades, what is the proposed fallback — e.g., online recalibration such as ACI, which already appears as a baseline?
5 comments
- Second, we add longer GRPO training and reward-over-optimization curves by extending Qwen3-8B training to 60 updates. The results illustrated below and Figure 4 in our revised manuscript show that safety scores peak at step 20 and decline afterward, even as predicted reward improves before leveling off. This divergence supports the selected checkpoint and provides an initial check against reward over-optimization. We have added the optimization curves and analysis to Appendix D.5.
| Step | 10 | 20 | 30 | 40 | 50 | 60 |
|---|---|---|---|---|---|---|
| Reward | -0.050 | -0.029 | -0.010 | -0.002 | -0.007 | -0.003 |
| Uncertainty | 0.115 | 0.094 | 0.070 | 0.066 | 0.069 | 0.049 |
| FFT-SS | 0.525 | 0.605 | 0.559 | 0.513 | 0.492 | 0.489 |
| WildGuardMix-SS | 0.707 | 0.795 | 0.759 | 0.745 | 0.719 | 0.710 |
- Finally, we add multi-seed runs with standard deviations using seeds 1, 2, and 42. The results below show that UARM achieves the highest mean safety score on all benchmarks, indicating that the observed advantage is not tied to a single seed. We have added these results and analysis to Appendix D.7.
| Method | HarmBench | FFT | DAN | WildGuardMix |
|---|---|---|---|---|
| Naive | 0.539$_{\pm 0.014}$ | 0.425$_{\pm 0.021}$ | 0.560$_{\pm 0.006}$ | 0.645$_{\pm 0.003}$ |
| MCNF | 0.610$_{\pm 0.015}$ | 0.524$_{\pm 0.011}$ | 0.612$_{\pm 0.005}$ | 0.723$_{\pm 0.006}$ |
| Clear | 0.574$_{\pm 0.008}$ | 0.510$_{\pm 0.009}$ | 0.603$_{\pm 0.012}$ | 0.707$_{\pm 0.013}$ |
| UARM | 0.661$_{\pm 0.006}$ | 0.613 | 0.689 | 0.780_{\pm 0.016}\ |
[W3,Q3] The @50 protocol is not comparable across methods, and full-set metrics are absent. Please provide full-test-set metrics (or metrics on one common fixed subset) for Table 1, risk-coverage and width-error curves, and a concrete explanation of the anomalously large PKU-SafeRLHF gap, including how the discrete severity labels and method-specific @50 subset selection contribute to it. Possible calibration/train leakage should be explicitly ruled out.
Response. Thank you for your meticulous and insightful comment. Our detailed actions and responses are organized as follows.
- First, we add full-test-set R$^2$, MSE, MAE, empirical coverage, mean interval width, and AURC on all three datasets. The results below reveal two observations: (i) UARM achieves the smallest mean width and lowest AURC, with coverage close to the nominal 90% target; (ii) its full-set prediction metrics are not uniformly best, whereas its @50 results are strongest on samples identified as most certain. This contrast supports accurate reward estimation on low-uncertainty samples, without attributing the gains solely to better ranking or full-set prediction quality. We have added these results and analysis to Appendix D.3.
| Dataset | Method | R$^2$ | MSE | MAE | Coverage | Efficiency | AURC |
|---|---|---|---|---|---|---|---|
| HelpSteer | Naive | 0.353 | 0.596 | 0.588 | - | - | - |
| MCNF | 0.426 | 0.529 | 0.559 | 0.916 | 2.191 | 0.496 | |
| Clear | 0.425 | 0.529 | 0.573 | 0.884 | 2.245 | 0.512 | |
| UARM | 0.411 | 0.542 | 0.517 | 0.901 | 2.064 | 0.465 | |
| UltraFeedback | Naive | 0.550 | 1.536 | 0.887 | - | - | - |
| MCNF | 0.668 | 1.134 | 0.759 | 0.947 | 3.857 | 0.550 | |
| Clear | 0.669 | 1.128 | 0.793 | 0.906 | 3.054 | 0.546 | |
| UARM | 0.665 | 1.141 | 0.747 | 0.905 | 2.911 | 0.514 | |
| PKU-SafeRLHF | Naive | 0.811 | 0.219 | 0.290 | - | - | - |
| MCNF | 0.836 | 0.189 | 0.242 | 0.959 | 1.092 | 0.083 | |
| Clear | 0.821 | 0.207 | 0.267 | 0.898 | 1.112 | 0.096 | |
| UARM | 0.827 | 0.200 | 0.172 | 0.902 | 0.561 | 0.044 |
- Second, we add risk-coverage and width-error curves below and in Figure 5 in our revised manuscript. The risk-coverage curves show that MSE generally increases as the retained coverage expands from the most certain samples to the full test set, whereas the width-error curves show a broadly positive association between interval width and MSE. Together, these trends support interval width as a useful indicator of reward-prediction reliability. We have added both curves and their analysis to Appendix D.3 and Figure 5(b-c).
| Dataset | 10% | 20% | 30% | 40% | 50% | 60% | 70% | 80% | 90% | 100% |
|---|---|---|---|---|---|---|---|---|---|---|
| HelpSteer | 0.171 | 0.273 | 0.311 | 0.325 | 0.387 | 0.376 | 0.393 | 0.416 | 0.456 | 0.542 |
| UltraFeedback | 0.178 | 0.202 | 0.260 | 0.305 | 0.383 | 0.404 | 0.450 | 0.575 | 0.741 | 1.139 |
| PKU-SafeRLHF | 0.003 | 0.005 | 0.002 | 0.006 | 0.013 | 0.016 | 0.048 | 0.083 | 0.116 | 0.200 |
| Dataset | Statistic | 1 | 2 | 3 | 4 | 5 | 6 | 7 | 8 | 9 | 10 | Avg |
|---|---|---|---|---|---|---|---|---|---|---|---|---|
| HelpSteer | Width | 1.324 | 1.779 | 1.971 | 2.035 | 2.065 | 2.099 | 2.171 | 2.284 | 2.373 | 2.541 | 2.064 |
| MSE | 0.383 | 0.441 | 0.472 | 0.504 | 0.535 | 0.504 | 0.598 | 0.629 | 0.664 | 0.695 | 0.542 | |
| UltraFeedback | Width | 1.696 | 1.981 | 2.186 | 2.381 | 2.565 | 2.775 | 3.059 | 3.465 | 4.020 | 4.999 | 2.913 |
| MSE | 0.210 | 0.273 | 0.584 | 0.666 | 0.723 | 0.783 | 0.893 | 1.897 | 2.145 | 3.215 | 1.139 | |
| PKU-SafeRLHF | Width | 0.001 | 0.002 | 0.002 | 0.016 | 0.086 | 0.238 | 0.767 | 1.047 | 1.410 | 2.039 | 0.561 |
| MSE | 0.021 | 0.009 | 0.009 | 0.017 | 0.031 | 0.101 | 0.287 | 0.355 | 0.403 | 0.772 | 0.200 |
- Finally, we clarify the performance gap across methods in PKU-SafeRLHF. On the full test set, UARM's MSE (0.200) is close to Naive (0.219), MCNF (0.189), and Clear (0.207). The much larger @50 differences arise on method-specific subsets, so they reflect both prediction quality and which samples each method selects; they should not be read as comparable full-set gains. The four discrete severity labels may also interact with quantile-based interval construction: UARM's narrowest deciles have widths as small as 0.001--0.002 and MSEs of 0.009--0.021, consistent with intervals concentrating near discrete ratings and selecting accurately predicted cases. There is no calibration/train leakage: training, calibration, and test sets are disjoint, and only training loss is used for model selection and early stopping.
[W4,Q4] The core assumptions of Eq. (14) are not empirically tested. Under GRPO the policy — and hence the reward distribution — drifts over update steps. Can the authors report empirical coverage stratified by interval width, reward magnitude, or training step over GRPO updates, together with PIT/QQ-style checks of the Gaussian-noise interpretation in Eq. (12)? If exchangeability degrades, what is the proposed fallback — e.g., online recalibration such as ACI, which already appears as a baseline?
Response. Thank you for your thoughtful and constructive comment. Our clarifications and actions are detailed as follows.
-
First, we agree that evaluating on-policy coverage and conducting PIT/QQ-style checks is important. However, these evaluations, including stratification by interval width, reward magnitude, or GRPO step, require independently annotated severity labels for generated rollouts. Such labels are unavailable, and annotation costs and rebuttal-time constraints prevented us from collecting them. Policy drift may therefore affect coverage, as acknowledged in our Limitations. Online recalibration, including ACI with newly labeled rollouts, is a promising direction for addressing this issue.
-
Second, we add PIT and QQ plots on the three offline test datasets to assess the Gaussian-noise interpretation in Eq. (12), as shown in Figure 7. The results reveal two observations: (i) PIT densities are broadly close to the uniform reference; (ii) QQ curves approximately follow over much of the range. These diagnostics provide qualified support for the local Gaussian approximation in Eq. (12), without establishing its validity under on-policy distribution shift. We have added the plots and analysis to Appendix D.12.
Thank you for your positive assessment of our empirical evaluation, theoretical analysis, computational efficiency, and presentation. Below are our responses to your specific concerns.
Response. Thank you for your thoughtful and insightful comment. Our detailed clarifications are organized as follows, and we have added these details to Appendix A.3.
Response. Thank you for your actionable and constructive comment. Our actions and responses are detailed as follows.