acceptodds
Under review as a conference paper at ICLR 2027

Not All Reward Matters: Uncertainty-Aware Reward Modeling for RLHF

Abstract

Reward models in reinforcement learning from human feedback (RLHF) distill human preferences into scalar signals that steer policy optimization, yet they offer no measure of evaluation confidence. Every input receives a single score, whether it is well-supported or ambiguous. Without uncertainty estimates, policy optimization methods such as Group Relative Policy Optimization (GRPO) treat all rewards as equally informative, propagating uncertain evaluations with the same weight as confident ones. To address this limitation, we propose **Uncertainty-Aware Reward Modeling (UARM)**, a framework that equips reward models with calibrated, per-sample uncertainty through quantile-based modeling and calibration. Each reward is represented as a point estimate and an adaptive prediction interval, enabling GRPO to downweight uncertain evaluations during advantage computation. Experiments demonstrate that UARM achieves higher reward prediction accuracy on uncertainty-ranked samples than baselines across three preference datasets, and delivers stronger downstream RLHF alignment. Code is available at https://anonymous.4open.science/r/UARM-42C0.

Primary area:
other topics in machine learning (i.e., none of the above)
Keywords:
Large Language Model, Reward Modeling, Reinforcement Learning from Human Feedback, Uncertainty Modeling

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.60 over 5 reviews, above 70% of the others.
Reviewer_MCAROpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
2
Soundness
3
Read review
Paper summary

The paper identifies the lack of confidence information in scalar reward models as a central, under-examined limitation of RLHF: GRPO standardizes rewards within each prompt group under the implicit assumption that all rewards are equally informative, which can overweight noisy, biased, or spuriously inflated scores. UARM addresses this as follows: (i) the scalar reward head is replaced by K+1K{+}1 jointly trained quantile heads optimized with pinball loss (Eq. 5), with the median serving as the point reward consumed by GRPO; (ii) adaptive prediction intervals are constructed by split-conformal calibration over the shortest contiguous block of nested interquantile intervals (Eqs. 7–9), which automatically widens in low-density regions; (iii) each interval half-width is converted into a per-sample observation noise variance under a local Gaussian approximation (Eq. 12), which is then subtracted from the group variance to recover the signal variance (Eq. 13); and (iv) GRPO advantages are reweighted by a sample-specific signal-to-noise attenuation factor σsignal2/(σsignal2+σnoise,i2)\sigma^2_{\text{signal}}/(\sigma^2_{\text{signal}} + \sigma^2_{\text{noise},i}) (Eq. 14), which reduces exactly to vanilla GRPO when noise is uniform across the group. UARM is evaluated on three preference datasets (HelpSteer, UltraFeedback, PKU-SafeRLHF) against a dozen comparators spanning the model-based UQ line (MC-Dropout, DER, Deep/Packed Ensembles, MCNF) and the distribution-free line (SCP, CQR, WCP, ACI, SCCP, CLEAR), across 8B–70B backbones (Qwen3-8B, LLaMA3.1-8B/70B), and on four safety-oriented downstream benchmarks (HarmBench, FFT, DAN, WildGuardMix) with two policy models.

Critical strengths
  1. Tight alignment between claims, evidence, and ablations. Table 2 cleanly disentangles the two components: removing conformal calibration (UARM†^\dagger) raises MSE by 21.7% on UltraFeedback (0.383 → 0.466), while replacing dense quantiles with three quantiles (UARM‡^\ddagger) reduces R2R^2 from 0.794 to 0.747. The full model is best on every dataset and metric, supporting the claimed synergy rather than a single-component story.
  2. Theoretically honest coverage analysis. Finite-sample marginal coverage (Theorem 3.1, Assumptions B.1/B.3) and asymptotic conditional coverage (Theorem 3.2, Assumptions B.2–B.4) are kept clearly separate; the conditional result states its required uniform quantile calibration, its deficit rate γ=4εNtrain+δNtrain+tNcal\gamma = 4\varepsilon_{N_{\text{train}}} + \delta_{N_{\text{train}}} + t_{N_{\text{cal}}}, and its failure regimes — notably, Remark B.7 explicitly states that the result applies to a fresh split and does not cover substantial policy-induced shift without covariate-shift correction. Marginal coverage is not overstated as conditional.
  3. Comprehensive baselines, consistent gains, negligible overhead. The benchmark suite covers essentially the full spectrum of modern UQ, and UARM achieves state-of-the-art uncertainty-ranked prediction on all three datasets, with gains that hold consistently across 8B–70B reward-model backbones (Table 3). Because the intervals are read directly from the calibrated quantile heads, no additional forward passes are required at inference — a meaningful advantage over ensemble methods.
  4. Clear and reproducible presentation. The Figure 1 case study is an effective intuitive anchor, and the reproduction details in Appendix C (score computation, training schedules, judge prompts) are thorough.
Critical weaknesses
  1. Missing highly related work: CFA (Conformal Feedback Alignment, arXiv:2601.17329). CFA already grounds preference weighting in conformal prediction: it constructs conformal prediction sets with controllable coverage to quantify answer-level reliability, aggregates set memberships into a per-pair weight via a set-wise aggregation rule, and injects that weight into both PPO-style reward-model training and the DPO preference loss. Given that Section A.3 explicitly positions UARM against uncertainty-aware policy-optimization work (ULPS, UPO, TUR-DPO), omitting the closest conformal-prediction-based alignment method is a substantive gap rather than a routine missing citation, and the axes of differentiation are currently left entirely to the reader.
  2. The "improves RLHF / code alignment" claim is established only on safety. All four downstream benchmarks measure jailbreak robustness or harmlessness; there is no helpfulness, instruction-following, or general-capability evaluation (e.g., AlpacaEval, MT-Bench, Arena-Hard, or knowledge/reasoning suites). Because the method downweights high-uncertainty rewards, and in a safety-oriented pipeline such rewards may coincide with responses that merely rephrase refusals, the current results cannot distinguish genuine alignment gains from a "safety by evasion" artifact, nor can they rule out degraded usefulness. The qualitative case study (Tables 5–6) shows the same pattern — refusals versus compliance — and does not close this gap.
  3. The @50 protocol is not comparable across methods, and full-set metrics are absent. Each uncertainty-aware method selects its own top-50% most uncertain samples (the Naive baseline instead uses a random 50% subset), so R2@50R^2@50/MSE@50 are computed on different subsets for every method; a method can look better partly by defining a subset on which it is accurate. Full-test-set metrics, metrics on a common fixed subset, and risk–coverage curves are needed for a fair comparison. In addition, the PKU-SafeRLHF result — R2@50=0.985R^2@50 = 0.985 vs. 0.850 (Naive) and 0.940 (CLEAR), with MSE shrinking from 0.173 to 0.013, a roughly 13×\times reduction — is a far larger jump than anything observed on HelpSteer or UltraFeedback, and possible causes (the discrete 0–3 severity labels interacting with interval construction, subset-selection effects, or calibration/train leakage) should be explicitly ruled out.
  4. The core assumptions of Eq. (14) are not empirically tested. The conversion of interval half-width to a Gaussian noise variance (Eq. 12) is a local Gaussian approximation, and the conformal step relies on exchangeability between calibration points and online rollouts. Under GRPO the policy — and hence the reward distribution — drifts over update steps; the paper acknowledges this only as future work (Section 5) and provides no coverage tracking across the 20 updates, no empirical check of the Gaussian approximation (e.g., residual or PIT diagnostics), and no comparison with online recalibration.
  5. Limited downstream experimental rigor. The 20-update budget is very short, Table 4 reports no multi-seed variance or error bars, and downstream fine-tuning uses only two 8B policy models despite the 8B–70B scaling evidence for the reward models. Effect sizes as large as the 35.09% relative improvement on FFT (LLaMA3.1-8B) cannot be assessed for stability without repeated runs.
Questions for authors
  1. Relation to CFA. Please state the exact differences along at least the following axes: (a) the object whose uncertainty is modeled — reward-model uncertainty over the scalar reward r(x,y)r(x,y) vs. answer-level reliability of the two responses in a preference pair; (b) the conformal construction — regression intervals from quantile heads via shortest contiguous nested blocks vs. membership of responses in two sets at different coverage levels, defined by NLL-based (white-box) or frequency/entropy/similarity-based (black-box) nonconformity scores; (c) where the weight enters the loss — an SNR attenuation factor on per-rollout GRPO advantages during online policy optimization, including the signal/noise variance decomposition of Eqs. 13–14, vs. a per-pair scalar weight multiplying the RM preference loss (PPO-style) or the DPO loss; and (d) granularity — per-rollout reward instances vs. per-preference-pair weights. Are the two approaches complementary, and could they be combined?
  2. Beyond safety. Please add (or commit to adding) helpfulness/general-capability evaluations alongside the safety scores, longer GRPO training beyond 20 updates with reward-over-optimization curves, and multi-seed runs with standard deviations for Table 4.
  3. Fair head-to-head metrics. Please provide full-test-set metrics (or metrics on one common fixed subset) for Table 1, risk–coverage and width–error curves, and a concrete explanation of the anomalously large PKU-SafeRLHF gap, including how the discrete severity labels and the method-specific @50 subset selection contribute to it.
  4. Validity under policy drift. Can the authors report empirical coverage stratified by interval width, reward magnitude, or training step over the course of GRPO updates, together with PIT/QQ-style checks of the Gaussian-noise interpretation in Eq. (12)? If exchangeability degrades, what is the proposed fallback — e.g., online recalibration such as ACI, which already appears as a baseline?
5 comments

Thank you for your positive assessment of our empirical evaluation, theoretical analysis, computational efficiency, and presentation. Below are our responses to your specific concerns.


[W1,Q1] Missing highly related work: CFA (Conformal Feedback Alignment, arXiv:2601.17329). CFA already grounds preference weighting in conformal prediction. Please state the exact differences along the following axes: (a) scalar reward uncertainty vs. answer-level reliability; (b) quantile regression intervals vs. two response sets at different coverage levels using NLL or frequency/entropy/similarity scores; (c) GRPO SNR attenuation and the variance decomposition of Eqs. 13-14 vs. RM/PPO-style or DPO loss weighting; and (d) per-rollout vs. per-preference-pair granularity. Are the two approaches complementary, and could they be combined?

Response. Thank you for your thoughtful and insightful comment. Our detailed clarifications are organized as follows, and we have added these details to Appendix A.3.

  • First, we clarify the uncertainty target. CFA estimates answer-level reliability for the two responses in a preference pair. UARM models the conditional distribution of a scalar reward for each prompt-response pair and derives uncertainty over the reward magnitude. It does not separately estimate answer correctness.
  • Second, we clarify the conformal constructions. CFA assigns reliability using membership in two response sets at different coverage levels, with NLL-based scores in the white-box setting or frequency, entropy, and similarity scores in the black-box setting. UARM instead trains quantile heads and calibrates shortest contiguous nested interquantile intervals for the scalar reward.
  • Third, we clarify where the weights enter optimization. CFA uses a per-pair weight in PPO-style reward-model loss or directly in the DPO loss. UARM uses interval-derived signal-to-noise attenuation on per-rollout GRPO advantages, motivated by the variance decomposition in Eqs. (13)--(14).
  • Finally, we clarify the granularity and complementarity. CFA operates on preference pairs, whereas UARM operates on individual rollout rewards. They are therefore potentially complementary. CFA-style answer-reliability weights could be combined with UARM's calibrated reward intervals and GRPO attenuation, but the combined method would require separate controls and is not evaluated here.

[W2,W5,Q2] The current results cannot distinguish genuine alignment gains from a "safety by evasion" artifact, nor can they rule out degraded usefulness. Downstream fine-tuning uses only two 8B policy models despite the 8B-70B scaling evidence for the reward models. Please add (or commit to adding) helpfulness/general-capability evaluations alongside the safety scores, longer GRPO training beyond 20 updates with reward-over-optimization curves, and multi-seed runs with standard deviations for Table 4.

Response. Thank you for your actionable and constructive comment. Our actions and responses are detailed as follows.

  • First, we add helpfulness and general response-quality evaluations using AlpacaEval and MT-Bench, together with ORBench-Hard and XSTest to assess over-refusal. The results below show that UARM achieves the best AlpacaEval win rate and MT-Bench score, while maintaining refusal rates comparable to or lower than MCNF and Clear. These results provide evidence against a broad usefulness trade-off in the evaluated setting. We have added these results and analysis to Appendix D.8.
MethodAlpacaEval-WRAlpacaEval-RIMTBench-ScoreMTBench-RIORBench-Hard-RRORBench-Hard-RIXSTest-RRXSTest-RI
Naive5.87%-5.206-20.70%-1.60%-
MCNF10.43%77.78%↑\uparrow5.82511.88%↑\uparrow15.69%24.18%↑\uparrow0.40%75.00%↑\uparrow
Clear10.43%77.78%↑\uparrow5.83112.00%↑\uparrow15.39%25.64%↑\uparrow0.80%50.00%↑\uparrow
UARM10.65%81.48%↑\uparrow5.84412.24%↑\uparrow15.01%27.47%↑\uparrow0.80%50.00%↑\uparrow
  • Second, we add longer GRPO training and reward-over-optimization curves by extending Qwen3-8B training to 60 updates. The results illustrated below and Figure 4 in our revised manuscript show that safety scores peak at step 20 and decline afterward, even as predicted reward improves before leveling off. This divergence supports the selected checkpoint and provides an initial check against reward over-optimization. We have added the optimization curves and analysis to Appendix D.5.
Step102030405060
Reward-0.050-0.029-0.010-0.002-0.007-0.003
Uncertainty0.1150.0940.0700.0660.0690.049
FFT-SS0.5250.6050.5590.5130.4920.489
WildGuardMix-SS0.7070.7950.7590.7450.7190.710
  • Finally, we add multi-seed runs with standard deviations using seeds 1, 2, and 42. The results below show that UARM achieves the highest mean safety score on all benchmarks, indicating that the observed advantage is not tied to a single seed. We have added these results and analysis to Appendix D.7.
MethodHarmBenchFFTDANWildGuardMix
Naive0.539$_{\pm 0.014}$0.425$_{\pm 0.021}$0.560$_{\pm 0.006}$0.645$_{\pm 0.003}$
MCNF0.610$_{\pm 0.015}$0.524$_{\pm 0.011}$0.612$_{\pm 0.005}$0.723$_{\pm 0.006}$
Clear0.574$_{\pm 0.008}$0.510$_{\pm 0.009}$0.603$_{\pm 0.012}$0.707$_{\pm 0.013}$
UARM0.661$_{\pm 0.006}$0.613±0.013_{\pm 0.013}0.689±0.005_{\pm 0.005}0.780_{\pm 0.016}\

[W3,Q3] The @50 protocol is not comparable across methods, and full-set metrics are absent. Please provide full-test-set metrics (or metrics on one common fixed subset) for Table 1, risk-coverage and width-error curves, and a concrete explanation of the anomalously large PKU-SafeRLHF gap, including how the discrete severity labels and method-specific @50 subset selection contribute to it. Possible calibration/train leakage should be explicitly ruled out.

Response. Thank you for your meticulous and insightful comment. Our detailed actions and responses are organized as follows.

  • First, we add full-test-set R$^2$, MSE, MAE, empirical coverage, mean interval width, and AURC on all three datasets. The results below reveal two observations: (i) UARM achieves the smallest mean width and lowest AURC, with coverage close to the nominal 90% target; (ii) its full-set prediction metrics are not uniformly best, whereas its @50 results are strongest on samples identified as most certain. This contrast supports accurate reward estimation on low-uncertainty samples, without attributing the gains solely to better ranking or full-set prediction quality. We have added these results and analysis to Appendix D.3.
DatasetMethodR$^2$MSEMAECoverageEfficiencyAURC
HelpSteerNaive0.3530.5960.588---
MCNF0.4260.5290.5590.9162.1910.496
Clear0.4250.5290.5730.8842.2450.512
UARM0.4110.5420.5170.9012.0640.465
UltraFeedbackNaive0.5501.5360.887---
MCNF0.6681.1340.7590.9473.8570.550
Clear0.6691.1280.7930.9063.0540.546
UARM0.6651.1410.7470.9052.9110.514
PKU-SafeRLHFNaive0.8110.2190.290---
MCNF0.8360.1890.2420.9591.0920.083
Clear0.8210.2070.2670.8981.1120.096
UARM0.8270.2000.1720.9020.5610.044
  • Second, we add risk-coverage and width-error curves below and in Figure 5 in our revised manuscript. The risk-coverage curves show that MSE generally increases as the retained coverage expands from the most certain samples to the full test set, whereas the width-error curves show a broadly positive association between interval width and MSE. Together, these trends support interval width as a useful indicator of reward-prediction reliability. We have added both curves and their analysis to Appendix D.3 and Figure 5(b-c).
Dataset10%20%30%40%50%60%70%80%90%100%
HelpSteer0.1710.2730.3110.3250.3870.3760.3930.4160.4560.542
UltraFeedback0.1780.2020.2600.3050.3830.4040.4500.5750.7411.139
PKU-SafeRLHF0.0030.0050.0020.0060.0130.0160.0480.0830.1160.200

DatasetStatistic12345678910Avg
HelpSteerWidth1.3241.7791.9712.0352.0652.0992.1712.2842.3732.5412.064
MSE0.3830.4410.4720.5040.5350.5040.5980.6290.6640.6950.542
UltraFeedbackWidth1.6961.9812.1862.3812.5652.7753.0593.4654.0204.9992.913
MSE0.2100.2730.5840.6660.7230.7830.8931.8972.1453.2151.139
PKU-SafeRLHFWidth0.0010.0020.0020.0160.0860.2380.7671.0471.4102.0390.561
MSE0.0210.0090.0090.0170.0310.1010.2870.3550.4030.7720.200
  • Finally, we clarify the performance gap across methods in PKU-SafeRLHF. On the full test set, UARM's MSE (0.200) is close to Naive (0.219), MCNF (0.189), and Clear (0.207). The much larger @50 differences arise on method-specific subsets, so they reflect both prediction quality and which samples each method selects; they should not be read as comparable full-set gains. The four discrete severity labels may also interact with quantile-based interval construction: UARM's narrowest deciles have widths as small as 0.001--0.002 and MSEs of 0.009--0.021, consistent with intervals concentrating near discrete ratings and selecting accurately predicted cases. There is no calibration/train leakage: training, calibration, and test sets are disjoint, and only training loss is used for model selection and early stopping.

[W4,Q4] The core assumptions of Eq. (14) are not empirically tested. Under GRPO the policy — and hence the reward distribution — drifts over update steps. Can the authors report empirical coverage stratified by interval width, reward magnitude, or training step over GRPO updates, together with PIT/QQ-style checks of the Gaussian-noise interpretation in Eq. (12)? If exchangeability degrades, what is the proposed fallback — e.g., online recalibration such as ACI, which already appears as a baseline?

Response. Thank you for your thoughtful and constructive comment. Our clarifications and actions are detailed as follows.

  • First, we agree that evaluating on-policy coverage and conducting PIT/QQ-style checks is important. However, these evaluations, including stratification by interval width, reward magnitude, or GRPO step, require independently annotated severity labels for generated rollouts. Such labels are unavailable, and annotation costs and rebuttal-time constraints prevented us from collecting them. Policy drift may therefore affect coverage, as acknowledged in our Limitations. Online recalibration, including ACI with newly labeled rollouts, is a promising direction for addressing this issue.

  • Second, we add PIT and QQ plots on the three offline test datasets to assess the Gaussian-noise interpretation in Eq. (12), as shown in Figure 7. The results reveal two observations: (i) PIT densities are broadly close to the uniform reference; (ii) QQ curves approximately follow y=xy=x over much of the range. These diagnostics provide qualified support for the local Gaussian approximation in Eq. (12), without establishing its validity under on-policy distribution shift. We have added the plots and analysis to Appendix D.12.

Reviewer_ZQk7OpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper proposes UARM, a reward model for RLHF that outputs both a reward score and an uncertainty estimate. It predicts 31 quantiles of the rating for each response and uses a held-out calibration set to turn them into a prediction interval, whose width serves as the uncertainty. In GRPO, responses with wider intervals get smaller advantages, so uncertain rewards influence the policy less. The authors show that UARM’s low-uncertainty predictions are more accurate than those of 12 uncertainty-estimation baselines, and that GRPO with UARM improves safety scores on four jailbreak benchmarks for Qwen3-8B and LLaMA3.1-8B.

Critical strengths
  1. The problem is relevant. GRPO’s standardization Ai=(ri−μ)/σA_i=(r_i-\mu)/\sigma treats all rewards as equally reliable, and UARM adds per-sample uncertainty at the cost of one forward pass plus post hoc calibration.

  2. The experimental scope is broad: 12 baselines, 3 datasets, 5 backbones and 4 safety benchmarks.

Critical weaknesses

W1. The uncertainty estimator is not new, and the new weighting rule is unjustified and does not fix the failure that motivates the paper.

(a) Novelty. Sec. 3.2 scores each label by the minimum number of consecutive interquantile intervals that cover it, then thresholds that number on a calibration set. This matches CIR (Guo et al., AAAI 2026), which the paper cites only in passing. CHR (Sesia & Romano, NeurIPS 2021) is not cited. Neither method is compared against.

(b) No derivation. The rule A~i=σsignal2σsignal2+σnoise,i2⋅ri−μσsignal,σsignal2=max⁡(0,σ2−σˉnoise2)+ζ\tilde A_i=\frac{\sigma^2_{\text{signal}}}{\sigma^2_{\text{signal}}+\sigma^2_{\text{noise},i}}\cdot\frac{r_i-\mu}{\sigma_{\text{signal}}},\qquad \sigma^2_{\text{signal}}=\max(0,\sigma^2-\bar\sigma^2_{\text{noise}})+\zeta is stated without derivation. The underlying variance decomposition requires reward errors to be uncorrelated with true rewards and independent within a group. Neither assumption is stated or checked.

(c) It cannot fix Fig. 1. The prefactor is positive, so A~i\tilde A_i keeps the sign of ri−μr_i-\mu, and μ\mu still includes the unreliable reward. In Fig. 1 with σnoise2=(1,1,1,40)\sigma^2_{\text{noise}}=(1,1,1,40), the advantages are (−0.17,−0.59,−1.09,+0.87)(-0.17,-0.59,-1.09,+0.87), compared with (−0.15,−0.53,−0.98,+1.66)(-0.15,-0.53,-0.98,+1.66) for standard GRPO. The verbose response is still the only one reinforced.

(d) Whole groups are silenced. If σ2≤σˉnoise2\sigma^2\le\bar\sigma^2_{\text{noise}}, every advantage in the group is ≈0. With 4 rollouts per prompt this happens often and largely at random: under Gaussian assumptions, about 43% of groups are affected when the signal and noise variances are equal. The gains may therefore come from suppressing low-spread groups rather than from per-sample weighting.

W2. The experiments show neither that the uncertainty identifies unreliable rewards in RLHF, nor that it causes the gains.

(a) Coverage is unsupported and unmeasured. The coverage guarantees require rollouts to be exchangeable with the calibration data. They are not, as the Limitations section acknowledges, and no empirical coverage is reported anywhere.

(b) Experiment 1 tests the wrong thing. It is in-distribution selective prediction, and each method uses its own point predictor. It tests neither calibration nor usefulness on policy rollouts.

(c) Experiment 2 lacks controls. It is the only RLHF evidence, yet it has no ablation separating UARM’s reward from its weighting, no existing uncertainty-aware RLHF baseline, no pre-training row, and only a single 20-step run.

(d) Safety cannot be told apart from refusal. The test sets contain only harmful prompts, the custom judge is unvalidated and counts refusals as safe, and the reward depends on severity only. Higher safety scores are therefore indistinguishable from more refusals.

Questions for authors

([E] = could change my assessment; [C] = clarification; [S] = suggestion)

[E] Please compare against CIR, CHR, and a categorical head over the rating levels. The labels take only 4–5 values.

[C] How is the weighting rule derived, and what value of ζ\zeta is used? Note also that σ2=1N∑i(ri−μ)2\sigma^2=\frac1N\sum_i(r_i-\mu)^2 underestimates the variance by a factor of (N−1)/N(N-1)/N.

[E] What fraction of groups end up with near-zero advantages? Please add three controls: one shared noise variance for all rollouts, widths shuffled within each group, and Dr. GRPO. Please also try more rollouts per prompt.

[S] Consider computing μ\mu with uncertainty weights, and compare against a pessimistic reward r^−λΔ\hat r-\lambda\Delta, where Δ\Delta is the interval half-width.

[E] Please report empirical coverage and interval width, both in-distribution and on policy rollouts, and show whether the width detects reward errors on rollouts.

[E] Please add: the UARM reward with standard GRPO, the ensemble penalty of Coste et al. (2023), the policy before training, and multiple seeds with confidence intervals.

[E] Please evaluate over-refusal and helpfulness, e.g., on the benign WildGuardTest prompts, XSTest, and a helpfulness win rate.

[C] Is the backbone frozen? How is severity mapped to the reward? Are the policies base or instruct models? Why use a custom judge instead of each benchmark’s official classifier?

[C] Does SCP produce constant-width intervals? Why is CQR’s MAE on PKU-SafeRLHF (0.305) worse than the Naive baseline evaluated on a random half (0.206)?

9 comments

Thank you for your recognition of our problem relevance, computational efficiency, and empirical evaluation. Below are our responses to your specific concerns.


[W1(a),Q1[E]] The uncertainty estimator matches CIR, which the paper cites only in passing; CHR is not cited. Please compare against CIR, CHR, and a categorical head over the rating levels. The labels take only 4-5 values.

Response. Thank you for your insightful and constructive comment. Our clarifications and actions are as follows.

  • First, we clarify the distinction between CIR, CHR, and UARM. CIR independently selects a globally shortest block of interquantile intervals for each block size, which does not in general guarantee a nested family and therefore does not directly support the same coverage argument. CHR instead estimates a conditional histogram and solves a shortest contiguous-bin problem under an explicit nesting constraint. UARM uses dense quantiles with greedy interval expansion to obtain nested intervals efficiently, and further uses their per-rollout uncertainty for GRPO advantage reweighting, which is outside the objectives of CIR and CHR.

  • Second, we add experiments comparing UARM with CIR, CHR, and the categorical head (denoted as CateHead). For CateHead, we train a softmax head over the observed rating levels, calibrate a probability-mass threshold, and take the enclosing interval of the selected levels. The results below show that UARM achieves the strongest accuracy across all datasets, which may reflect its dense quantile modeling and calibrated nested intervals. We have added these more baseline comparisons and analysis to Appendix D.10.

MethodHelpSteer-R$^2$HelpSteer-MSEHelpSteer-MAEUltraFeedback-R$^2$UltraFeedback-MSEUltraFeedback-MAEPKU-SafeRLHF-R$^2$PKU-SafeRLHF-MSEPKU-SafeRLHF-MAE
Naive0.3570.6110.5950.5631.4810.8320.8500.1730.206
CateHead0.4380.5040.4190.6670.4540.4620.8820.1410.113
CHR0.4830.4870.5370.6720.4730.5540.9050.0590.076
CIR0.5230.4480.4830.7540.5370.5790.9440.0390.042
UARM0.5430.3870.4230.7940.3830.4610.9850.0130.016

[W1(b),Q2[C]] How is the weighting rule derived, and what value of ζ\zeta is used? The variance decomposition requires reward errors to be uncorrelated with true rewards and independent within a group. Neither assumption is stated or checked. Note also that σ2=1N∑i(ri−μ)2\sigma^2=\frac{1}{\mathrm{N}}\sum_i(r_i-\mu)^2 underestimates the variance by a factor of (N−1)/N(\mathrm{N}-1)/\mathrm{N}.

Response. Thank you for your observant and helpful comment. Our clarifications are as follows.

  • First, we clarify the derivation of the weighting rule. For R=R′+εiR=R'+\varepsilon_i with centered latent reward R′R', zero-mean noise and variances σsignal2\sigma_{\mathrm{signal}}^2 and σnoise,i2\sigma_{\mathrm{noise},i}^2, minimizing E[(R′−wiR)2]\mathbb{E}[(R'-w_iR)^2] gives wi=σsignal2/(σsignal2+σnoise,i2)w_i=\sigma_{\mathrm{signal}}^2/(\sigma_{\mathrm{signal}}^2+\sigma_{\mathrm{noise},i}^2). Normalizing by σsignal\sigma_{\mathrm{signal}} yields Eq. (14) when RR is approximated by the centered reward ri−μr_i-\mu. This provides a scalar shrinkage motivation under explicit assumptions. We have added this derivation to Appendix B.4.

  • Second, we clarify the choice of ζ\zeta. We use ζ=10−6\zeta=10^{-6} as a numerical stabilizer to avoid division by zero when the estimated signal variance is very small. We have added this numerical setting to Appendix C.2.

  • Finally, we clarify the variance convention. The expression σ2=1N∑i(ri−μ)2\sigma^2=\frac{1}{\mathrm{N}}\sum_i(r_i-\mu)^2 is a common formulation for within-group normalization. In our implementation, however, we use PyTorch's torch.std function with its default unbiased setting, which estimates the sample variance with denominator N−1\mathrm{N}-1. This only changes the normalization scale and does not change the within-group reward ordering.

[W1(c),Q4[S]] It cannot fix Fig. 1. The prefactor is positive, so A~i\tilde{A}_i keeps the sign of ri−μr_i-\mu, and μ\mu still includes the unreliable reward. The verbose response is still the only one reinforced. Consider computing μ\mu with uncertainty weights, and compare against a pessimistic reward r^−λΔ\hat{r}-\lambda\Delta, where Δ\Delta is the interval half-width.

Response. Thank you for your insightful and meticulous suggestion. We agree that Eq. (14) attenuates the contribution of uncertain rewards but does not by itself restore their ordering. Since the attenuation factor is positive, the resulting advantage preserves the sign of ri−μr_i-\mu, and the unweighted group mean can still be affected by an unreliable extreme reward. A pessimistic reward of the form r^i−λΔi\hat r_i-\lambda\Delta_i is a promising alternative because it modifies the reward before standardization and can change the resulting advantage sign. In this work, we focus on uncertainty-based attenuation, while leaving this complementary reward-shaping strategy, including the choice of λ\lambda, for future investigation. We have further clarified this scope in Section 3.2 and added pessimistic reward shaping as a promising direction in the future work.

[W1(d),Q3[E]] The gains may therefore come from suppressing low-spread groups rather than from per-sample weighting. What fraction of groups end up with near-zero advantages? Please add three controls: one shared noise variance for all rollouts, widths shuffled within each group, and Dr. GRPO. Please also try more rollouts per prompt.

Response. Thank you for your actionable and constructive comment. Our actions and responses are as follows.

  • First, we report the fraction of groups with near-zero advantages. For each prompt group gg, we compute the RMS of its scalar advantages A~g,i\tilde A_{g,i} and classify the group as near-zero if its RMS is at most 10−310^{-3}. The results below show an average of 16.75 out of 256 groups per update (6.54%), indicating that near-zero advantages affect a small fraction of groups in our training runs.
Step1234567891011121314151617181920Avg
Count141719161620222216132012151218211420131516.75
Percentage5.47%6.64%7.42%6.25%6.25%7.81%8.59%8.59%6.25%5.08%7.81%4.69%5.86%4.69%7.03%8.20%5.47%7.81%5.08%5.86%6.54%
  • Second, we add experiments with widths shuffled within each group (†^\dagger), one shared noise variance (‡^\ddagger), and Dr. GRPO (§^\S) on Qwen3-8B. The results below reveal two observations: (i) shuffling widths yields scores relatively close to Naive, highlighting the importance of correctly matching uncertainty to individual rollouts; (ii) UARM outperforms both shared-noise weighting and Dr. GRPO on all benchmarks, supporting an additional benefit from per-rollout uncertainty beyond group-level scaling or normalization changes alone. We have added these results and analysis to Appendix D.11.
MethodHarmBench-SSHarmBench-RIFFT-SSFFT-RIDAN-SSDAN-RIWildGuardMix-SSWildGuardMix-RI
Naive0.540-0.445-0.555-0.642-
UARM$^\dagger$0.5532.41%↑\uparrow0.4511.28%↑\uparrow0.5997.82%↑\uparrow0.6602.74%↑\uparrow
UARM$^\ddagger$0.5919.35%↑\uparrow0.55925.62%↑\uparrow0.66820.24%↑\uparrow0.6988.67%↑\uparrow
UARM$^\S$0.63818.15%↑\uparrow0.58431.20%↑\uparrow0.66820.37%↑\uparrow0.72112.33%↑\uparrow
UARM0.66423.06%↑\uparrow0.60535.90%↑\uparrow0.69525.13%↑\uparrow0.79523.75%↑\uparrow
  • Third, we add experiments varying the number of rollouts to assess sensitivity to larger group size. The results below show that safety scores generally improve with larger groups. UARM exceeds the Naive reference at every tested group size, indicating that the gains are not confined to small rollouts. We have added these sensitivity results and analysis to Appendix D.6.
Nrol\mathrm{N}_\mathrm{rol}HarmBench-SSHarmBench-RIFFT-SSFFT-RIDAN-SSDAN-RIWildGuardMix-SSWildGuardMix-RI
Naive0.540-0.445-0.555-0.642-
40.66423.06%↑\uparrow0.60535.90%↑\uparrow0.69525.13%↑\uparrow0.79523.75%↑\uparrow
80.68426.76%↑\uparrow0.61237.60%↑\uparrow0.71328.33%↑\uparrow0.78321.92%↑\uparrow
120.70730.93%↑\uparrow0.62239.74%↑\uparrow0.72129.86%↑\uparrow0.79824.21%↑\uparrow
160.70931.30%↑\uparrow0.63141.87%↑\uparrow0.72630.76%↑\uparrow0.80625.58%↑\uparrow

[W2(a,b),Q5[E]] The coverage guarantees require rollouts to be exchangeable with the calibration data. They are not. Experiment 1 is in-distribution selective prediction, and each method uses its own point predictor; it tests neither calibration nor usefulness on policy rollouts. Please report empirical coverage and interval width, both in-distribution and on policy rollouts, and show whether the width detects reward errors on rollouts.

Response. Thank you for your meticulous and insightful comment. Our actions and responses are as follows.

  • First, we add evaluation of empirical coverage and mean interval width on in-distribution test sets. The results below show that UARM achieves coverage close to the nominal 90% target, with the smallest mean interval width and lowest AURC among the evaluated uncertainty-aware methods. These results support sharp intervals with near-target empirical calibration and lower selective risk on the labeled test sets. We have added the results and analysis to Appendix D.3.
DatasetMethodCoverageEfficiencyAURC
HelpSteerMCNF0.9162.1910.496
Clear0.8842.2450.512
UARM0.9012.0640.465
UltraFeedbackMCNF0.9473.8570.550
Clear0.9063.0540.546
UARM0.9052.9110.514
PKU-SafeRLHFMCNF0.9591.0920.083
Clear0.8981.1120.096
UARM0.9020.5610.044
  • Second, we agree that evaluation on policy rollouts is important. However, the generated rollouts lack independently annotated severity labels on the reward model's target scale, so coverage and reward-error detection cannot be measured from the available safety judgments. Due to the annotation cost and rebuttal-time constraints, we did not conduct this rollout-level evaluation.
Reviewer_krFAOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper identifies the absence of uncertainty estimates in RLHF reward models as a problem that propagates into policy optimization: GRPO standardizes all rewards within a rollout group equally, so an unreliable extreme score can dominate the advantage computation. The authors propose UARM (Uncertainty-Aware Reward Modeling), which replaces the standard point-estimate reward model with a quantile regressor that outputs K+1 conditional quantiles, trained jointly via pinball loss. A conformal-style post-hoc calibration step on a held-out set produces adaptive prediction intervals whose width serves as a per-sample uncertainty measure. Two theoretical results provide marginal coverage (Theorem 3.1, finite-sample) and asymptotic conditional coverage (Theorem 3.2). The interval width is converted into an observation-noise variance estimate and used to reweight GRPO advantages via a signal-to-noise attenuation factor (Eq. 14), downweighting rollouts with wide intervals. Experiments on three preference datasets (HelpSteer, UltraFeedback, PKU-SafeRLHF) compare against 12 uncertainty-quantification baselines on uncertainty-ranked reward prediction (R²@50, MSE@50, MAE@50). Ablations isolate the contributions of multi-quantile modeling and calibration. Generalization is tested across Qwen3 and LLaMA3.1 backbones (8B–70B). Downstream RLHF experiments fine-tune policies via GRPO with uncertainty-weighted advantages and evaluate on four safety benchmarks (HarmBench, FFT, DAN, WildGuardMix), showing consistent safety-score improvements over baselines.

Critical strengths
  1. Well-motivated, practically relevant problem. The observation that conventional reward models provide no uncertainty signal and that GRPO's uniform standardization amplifies unreliable scores is correct, clearly demonstrated (Figure 1), and consequential for deployed RLHF systems. The paper identifies a genuine gap in the standard RLHF pipeline and proposes a practical solution that requires only a lightweight MLP head and a calibration set — no ensemble inference or multiple forward passes at deployment time. This efficiency advantage over model-based UQ methods is practically important.

  2. Thorough experimental comparison and clean ablation. The comparison against 12 baselines spanning both model-based and distribution-free UQ methods, across three datasets and multiple model scales (8B–70B), is comprehensive. The ablation (Table 2) cleanly isolates the complementary contributions of multi-quantile modeling and calibration. The hyperparameter sensitivity analysis (Figure 3) shows stability across a range of (α, K) configurations. This experimental thoroughness strengthens confidence in the empirical claims.

Critical weaknesses
  1. Limited methodological novelty. UARM combines three well-known components: quantile regression (Steinwart & Christmann, 2011), conformal-style calibration with nested intervals, and variance-based reweighting. The contribution is their application to reward modeling in RLHF. While the application is sensible and well-executed, no component is new. The coverage guarantees (Theorems 3.1, 3.2) are direct applications of existing conformal prediction theory. The advantage-reweighting formula (Eq. 14) is a heuristic variance decomposition. For a top venue, the paper would need either novel methodology or substantially deeper analysis of why this combination works in the RLHF context — e.g., characterizing when uncertainty-aware reweighting provably improves policy optimization, or analyzing the interaction between reward uncertainty and GRPO's standardization.

  2. Incomplete downstream evaluation undermines the main claim. The paper's title ("Not All Reward Matters") and framing promise that uncertainty-aware reward modeling improves RLHF alignment broadly. Yet the downstream evaluation (Table 4) measures only safety scores on adversarial jailbreak benchmarks. This is a narrow slice of alignment. The paper does not evaluate helpfulness, instruction-following quality, coherence, or any general-capability metric. Since the reweighting mechanism attenuates uncertain rollouts, it could systematically downweight creative, unusual, or exploratory responses that are genuinely high-quality but hard to evaluate — precisely the responses that RLHF should encourage. Without evidence that helpfulness is preserved, the safety improvements may come at an unreported cost, and the paper's central claim remains only partially validated.

Questions for authors
  1. Does uncertainty-aware advantage reweighting degrade helpfulness or general response quality? Please report standard helpfulness/quality metrics (e.g., MT-Bench, AlpacaEval, or the helpfulness ratings already available in HelpSteer) for the GRPO-tuned policies. This is the most important missing evaluation.

  2. How well-calibrated are UARM's intervals across uncertainty levels? The @50 metrics show accuracy on low-uncertainty samples, but do not assess calibration. Can you report empirical coverage by uncertainty decile (i.e., does the claimed 90% coverage hold separately for high-uncertainty and low-uncertainty samples)?

  3. The variance decomposition (Eq. 13) assumes ri=ri∗+εir_i = r_i^* + \varepsilon_i with the noise variance estimated from the interval width. How sensitive are the downstream results to the Gaussianity assumption? Have you tried simpler reweighting schemes (e.g., directly using 1/∣I(xi)∣1/|\mathcal{I}(x_i)| as a weight) that avoid this assumption?

  4. The quantile model is trained with pinball loss across K+1 quantile levels, which provides a richer supervision signal than the MSE loss used by the Naive baseline. To what extent do the R²@50 improvements reflect this richer training objective rather than the uncertainty estimation per se? A fair comparison would train the Naive baseline with the same architecture and total loss computation.

5 comments

Thank you for your recognition of our motivation, practical utility, and empirical evaluation. Below are our responses to your specific concerns.


[W1] Limited methodological novelty. UARM combines three well-known components: quantile regression, conformal-style calibration with nested intervals, and variance-based reweighting. The coverage guarantees are direct applications of existing conformal prediction theory, and Eq. (14) is a heuristic variance decomposition. The paper would need either novel methodology or substantially deeper analysis of why this combination works in RLHF — e.g., characterizing when reweighting provably improves policy optimization, or analyzing the interaction between reward uncertainty and GRPO's standardization.

Response. Thank you for your insightful comment. Our detailed responses and clarifications are as follows.

  • First, we clarify the scope of UARM's methodological contribution. Quantile regression and conformal calibration are established tools. However, without adapting them to online GRPO rollouts and group-relative advantages, these tools do not directly provide a per-rollout reliability signal for policy updates. UARM makes this connection explicit: the median provides the reward, while the calibrated interval provides the reliability signal for downstream GRPO updates.

  • Second, we clarify Theorem 3.1's relation to standard conformal prediction. Its calibration argument follows the standard calibration-set principle, while our dense quantile construction requires the stated nestedness assumption so that enlarging an interval never excludes a previously covered reward. This is a scope condition for applying the guarantee, rather than a new conformal theorem.

  • Finally, we clarify the statistical rationale for Eq. (14) reweighting. If observed reward R=R′+εiR=R'+\varepsilon_i, with centered latent reward R′R', zero-mean noise εi\varepsilon_i uncorrelated with R′R', and variances σsignal2\sigma_{\mathrm{signal}}^2 and σnoise,i2\sigma_{\mathrm{noise},i}^2, then for a scalar estimate wRwR,

    E[(R′−wR)2]=(1−w)2σsignal2+w2σnoise,i2,\mathbb E[(R'-wR)^2]=(1-w)^2\sigma_{\mathrm{signal}}^2+w^2\sigma_{\mathrm{noise},i}^2,

    which is minimized by bounded factor w=σsignal2/(σsignal2+σnoise,i2)w=\sigma_{\mathrm{signal}}^2/(\sigma_{\mathrm{signal}}^2+\sigma_{\mathrm{noise},i}^2). This derivation motivates Eq. (14) GRPO reweighting under the stated assumptions. We have added this analysis to Appendix B.

[W2,Q1] The downstream evaluation measures only safety scores on adversarial jailbreak benchmarks. Reweighting could systematically downweight creative, unusual, or exploratory responses that are genuinely high-quality but hard to evaluate. Does uncertainty-aware advantage reweighting degrade helpfulness or general response quality? Please report standard helpfulness/quality metrics (e.g., MT-Bench, AlpacaEval, or the helpfulness ratings already available in HelpSteer) for the GRPO-tuned policies.

Response. Thank you for your constructive and meticulous suggestion. Our detailed actions and responses are as follows.

  • Additional Experiments. We add evaluations on AlpacaEval and MT-Bench with win rate and win score to assess helpfulness and general response quality, and on XSTest and ORBench-Hard with refusal rate to assess over-refusal, comparing UARM with Naive, MCNF, and Clear.
MethodAlpacaEval-WRAlpacaEval-RIMTBench-ScoreMTBench-RIORBench-Hard-RRORBench-Hard-RIXSTest-RRXSTest-RI
Naive5.87%-5.206-20.70%-1.60%-
MCNF10.43%77.78%↑\uparrow5.82511.88%↑\uparrow15.69%24.18%↑\uparrow0.40%75.00%↑\uparrow
Clear10.43%77.78%↑\uparrow5.83112.00%↑\uparrow15.39%25.64%↑\uparrow0.80%50.00%↑\uparrow
UARM10.65%81.48%↑\uparrow5.84412.24%↑\uparrow15.01%27.47%↑\uparrow0.80%50.00%↑\uparrow
  • Result analysis. There are two observations. (i) UARM's stronger safety performance is accompanied by comparable or better helpfulness and over-refusal results relative to MCNF and Clear. (ii) All three uncertainty-aware methods improve helpfulness and reduce over-refusal relative to Naive. These findings support safety gains without a broad usefulness trade-off on the evaluated benchmarks.

  • Revision. We have added the above results and analysis to Appendix D.8.

[Q2] How well-calibrated are UARM's intervals across uncertainty levels? The @50 metrics show accuracy on low-uncertainty samples, but do not assess calibration. Can you report empirical coverage by uncertainty decile (i.e., does the claimed 90% coverage hold separately for high-uncertainty and low-uncertainty samples)?

Response. Thank you for your helpful and constructive suggestion. Our actions and responses are as follows.

  • Additional Experiments. We add UARM's empirical coverage by interval-width decile on all three test sets and report the mean coverage in table below. Samples are grouped by increasing interval width without using their labels.
Dataset12345678910Avg
HelpSteer0.9100.9030.8960.8890.9510.9510.9170.8610.8540.8810.901
UltraFeedback0.9410.9500.9250.9160.9060.8970.9090.8780.8690.8560.905
PKU-SafeRLHF0.9300.9920.9920.8560.9090.8900.8280.8810.9070.8390.902
  • Result analysis. UARM's decile-wise coverage remains generally close to the nominal 90% target across all three datasets, and the mean coverage is consistent with the marginal guarantee in Theorem 3.1.

  • Revision. We have added the average coverage and interval width, together with the decile coverage curve and analysis to Appendix D.3 and Figure 5(a).

[Q3] The variance decomposition (Eq. 13) assumes ri=ri∗+εir_i=r_i^* + \varepsilon_i, with the noise variance estimated from the interval width. How sensitive are the downstream results to the Gaussianity assumption? Have you tried simpler reweighting schemes (e.g., directly using 1/∣I(xi)∣1/|\mathcal I(x_i)| as a weight) that avoid this assumption?

Response. Thank you for your actionable and meticulous suggestion. Our responses and actions are as follows.

  • First, we clarify the role of the Gaussianity assumption. As our response to your concern in [W1], the MSE argument motivates the bounded attenuation in Eq. (14), while local Gaussianity is used only to map interval width to σnoise,i2\sigma_{\mathrm{noise},i}^2 through Eq. (12). Thus, it motivates our particular weighting but is not required for uncertainty reweighting. Inverse-interval weighting and other uncertainty-based mappings remain viable. We have added the derivation to Appendix B.

  • Second, we add experiments with inverse-interval weighting to assess sensitivity to the Gaussianity assumption. The results below show that this alternative improves safety over Naive on all configurations, but scores generally below the Gaussian-based weighting. Thus, the downstream gains persist without the Gaussian width-to-variance mapping, while their magnitude depends on the weighting paradigm. The gap may reflect that direct inverse-width weighting omits group-level signal variance and the bounded signal-to-noise shrinkage in Eq. (14). We have added these results and analysis to Appendix D.9.

MethodHarmBench-SSHarmBench-RIFFT-SSFFT-RIDAN-SSDAN-RIWildGuardMix-SSWildGuardMix-RI
Naive0.540-0.445-0.555-0.642-
MCNF$^\dagger$0.5868.52%↑\uparrow0.49411.10%↑\uparrow0.5875.64%↑\uparrow0.6866.85%↑\uparrow
MCNF0.61313.52%↑\uparrow0.53219.64%↑\uparrow0.61811.28%↑\uparrow0.73013.70%↑\uparrow
Clear$^\dagger$0.5685.09%↑\uparrow0.4715.98%↑\uparrow0.5834.99%↑\uparrow0.6745.03%↑\uparrow
Clear0.5848.06%↑\uparrow0.50814.09%↑\uparrow0.6028.45%↑\uparrow0.71911.88%↑\uparrow
UARM$^\dagger$0.62916.48%↑\uparrow0.54021.35%↑\uparrow0.61811.28%↑\uparrow0.71911.88%↑\uparrow
UARM0.66423.06%↑\uparrow0.60535.90%↑\uparrow0.69525.13%↑\uparrow0.79523.75%↑\uparrow

[Q4] The quantile model is trained with pinball loss across K+1 quantile levels, which provides a richer supervision signal than the MSE loss used by the Naive baseline. To what extent do the R$^2$@50 improvements reflect this richer training objective rather than the uncertainty estimation per se? A fair comparison would train the Naive baseline with the same architecture and total loss computation.

Response. Thank you for your insightful comment. Our detailed clarifications and responses are as follows.

  • First, we agree that the @50 metrics gains cannot by themselves quantify the contribution of the richer pinball objective versus uncertainty estimation. However, a point-prediction Naive model cannot use the same loss over K+1\mathrm{K}+1 quantiles. Giving it quantile outputs and this loss would instead make it a quantile model, not the original MSE-trained baseline.

  • Second, to better distinguish the effects of the training objective and uncertainty-based selection, we add full-set (@100) and five-run random-half (@r50) evaluations for all methods in below table, complementing the uncertainty-selected half (@50) evaluation. UARM's @r50 metrics remain close to @100, while uncertainty-based @50 improves all metrics on each dataset. These results indicate that UARM's gains combine the quality of its quantile-based predictor with an additional benefit from uncertainty-based selection, as evidenced by the gap between @r50 and @50. We have added these results and analysis to Appendix D.3.

DatasetMethodR$^2$@100MSE@100MAE@100R$^2$@r50MSE@r50MAE@r50R$^2$@50MSE@50MAE@50
HelpSteerNaive0.3530.5960.5880.3570.6110.595---
UARM0.4110.5420.5170.4100.5610.5250.5430.3870.423
UltraFeedbackNaive0.5501.5360.8870.5631.4810.832---
UARM0.6651.1410.7470.6611.1480.7450.7940.3830.461
PKU-SafeRLHFNaive0.8110.2190.2900.8500.1730.206---
UARM0.8270.2000.1720.8260.2000.1720.9850.0130.016
Reviewer_2djcOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper argues that reward models in RLHF give a single ambiguous score for every response without saying how confident they are, and that GRPO then treats confident and uncertain rewards the same way. The proposed method UARM trains a small head on top of an existing reward model to predict a set of reward quantiles instead of one score. The median serves as the reward, and the spread of the quantiles, calibrated post hoc on a held-out set, gives a per-sample prediction interval. The authors prove a finite-sample marginal coverage guarantee and an asymptotic conditional one. They then use the interval width in GRPO, shrinking the advantage of rollouts whose rewards are more uncertain. Experiments on HelpSteer, UltraFeedback, and PKU-SafeRLHF show that UARM’s least uncertain predictions are more accurate than those of many uncertainty baselines, and GRPO training with UARM’s reweighting gives higher safety scores on four jailbreak benchmarks for two policy models.

Critical strengths
  • Clear motivation and writing. The problem of reward models giving no sense of confidence is easy to understand, and the paper explains it well, including a simple example of how one inflated reward can distort GRPO’s group-normalized advantages.
  • Simple and cheap method. UARM adds only a small MLP head to an existing reward model and requires no extra forward passes or ensembles. The change to GRPO is a single reweighting formula, so the method is easy to adopt.
  • Broad comparison. The paper compares against a large set of uncertainty baselines, both model-based and conformal, across three datasets and several backbones ranging from 8B to 70B.
Critical weaknesses
  • The evaluation does not test the main claim. The paper’s selling point is calibrated intervals with coverage guarantees, yet it never reports empirical coverage or interval width, so readers cannot tell whether the intervals are actually calibrated. The main metric R² on the half of samples with the lowest uncertainty, mixes up two things: how good each method’s point prediction is, and how well it ranks uncertainty. UARM’s gains may simply come from a better median estimate. The Naive baseline is evaluated on a random half instead, which is not directly comparable.
  • The RLHF experiment cannot show where the gain comes from. There is no run that uses UARM’s median reward with standard GRPO (no reweighting), so it is unclear whether the improvement comes from the reward model itself or from the uncertainty weighting. Only safety (refusal) is measured, without any check of helpfulness or over-refusal, so a policy that refuses more often would also score higher. Training is also very short, with a single judge and a single run, which makes the large reported gains hard to interpret.
  • Predicting quantiles and then calibrating the interval on a held-out set is close to existing distribution-free methods such as CQR, which the paper itself uses as a baseline. Theorem 3.1 follows from the standard calibration-set argument, and the GRPO reweighting is essentially inverse-variance weighting. More importantly, the coverage guarantee assumes the calibration data and test samples come from the same distribution, but during RLHF the policy keeps changing, so the guarantee does not hold exactly where the method is meant to be used; the authors acknowledge this in their limitations.
Questions for authors
  1. What are the empirical coverage and average interval width of UARM and the baselines on each test set, and how does coverage hold up on policy rollouts during GRPO training?
  2. How does GRPO perform with UARM’s median reward but no uncertainty reweighting?
5 comments

Thank you for your positive feedback and appreciation of our motivation, practical utility, empirical evaluation, and presentation. Below are our responses to your specific concerns.


[W1,Q1] The evaluation does not test the main claim: it never reports empirical coverage or interval width, so readers cannot tell whether the intervals are actually calibrated. R$^2$ on the half of samples with the lowest uncertainty mixes up point prediction and uncertainty ranking. UARM's gains may simply come from a better median estimate. The Naive baseline is evaluated on a random half instead, which is not directly comparable. What are the empirical coverage and average interval width of UARM and the baselines on each test set, and how does coverage hold up on policy rollouts during GRPO training?

Response. Thank you for your actionable and constructive comment. Our responses and actions are organized as follows.

  • First, we supplement the evaluation to report empirical coverage and mean interval width, together with additional full-set R$^2$/MSE/MAE and AURC in the table below. We have two observations: (i) UARM has the smallest mean width and lowest AURC, with coverage close to the nominal 90% target, supporting sharp intervals with near-target empirical calibration and lower selective risk across retention levels; (ii) its median predictions do not consistently lead in full-set accuracy, yet its @50 results are strongest on samples identified as most certain, supporting accurate estimation on low-uncertainty samples rather than uniformly superior median predictions. We have added the complementary evaluation results and analysis to Appendix D.3.
DatasetMethodR$^2$MSEMAECoverageEfficiencyAURC
HelpSteerNaive0.3530.5960.588---
MCNF0.4260.5290.5590.9162.1910.496
Clear0.4250.5290.5730.8842.2450.512
UARM0.4110.5420.5170.9012.0640.465
UltraFeedbackNaive0.5501.5360.887---
MCNF0.6681.1340.7590.9473.8570.550
Clear0.6691.1280.7930.9063.0540.546
UARM0.6651.1410.7470.9052.9110.514
PKU-SafeRLHFNaive0.8110.2190.290---
MCNF0.8360.1890.2420.9591.0920.083
Clear0.8210.2070.2670.8981.1120.096
UARM0.8270.2000.1720.9020.5610.044
  • Second, we clarify when coverage holds on policy rollouts during GRPO training. Theorem 3.1 requires exchangeability between calibration data and new rollout-reward pairs, conditional on the fitted model, and nested intervals. Theorem 3.2 adds independent i.i.d. splits and uniform quantile calibration. As noted in our Limitations, policy-induced shift may invalidate these conditions and degrade coverage. Adaptive conformal prediction or semi-supervised reward model re-training using retained high-confidence rollouts offers promising avenues for addressing this drift.

[W2,Q2] The RLHF experiment cannot show where the gain comes from. How does GRPO perform with UARM's median reward but no uncertainty reweighting? Only safety (refusal) is measured, without any check of helpfulness or over-refusal, so a policy that refuses more often would also score higher. Training is also very short, with a single judge and a single run, which makes the large reported gains hard to interpret.

Response. Thank you for your insightful and meticulous comment. Our actions and responses are detailed below.

  • First, we add GRPO experiments using the same median reward with no uncertainty reweighting (†^\dagger) to separate reward quality from the contribution of weighting. The results below show improved safety with reweighting on all benchmarks for MCNF, Clear, and UARM. UARM's reward-only performance is comparable to the other variants, whereas full UARM performs best, supporting gains beyond median-reward quality alone. We have added the results and analysis to Appendix D.4.
MethodHarmBench-SSHarmBench-RIFFT-SSFFT-RIDAN-SSDAN-RIWildGuardMix-SSWildGuardMix-RI
Naive0.540-0.445-0.555-0.642-
MCNF$^\dagger$0.5736.11%↑\uparrow0.4736.41%↑\uparrow0.5824.74%↑\uparrow0.6694.11%↑\uparrow
MCNF0.61313.52%↑\uparrow0.53219.64%↑\uparrow0.61811.28%↑\uparrow0.73013.70%↑\uparrow
Clear$^\dagger$0.5573.15%↑\uparrow0.4623.84%↑\uparrow0.5773.84%↑\uparrow0.6572.29%↑\uparrow
Clear0.5848.06%↑\uparrow0.50814.09%↑\uparrow0.6028.45%↑\uparrow0.71911.88%↑\uparrow
UARM$^\dagger$0.5715.74%↑\uparrow0.4715.98%↑\uparrow0.5743.46%↑\uparrow0.6775.48%↑\uparrow
UARM0.66423.06%↑\uparrow0.60535.90%↑\uparrow0.69525.13%↑\uparrow0.79523.75%↑\uparrow
  • Second, we add helpfulness (AlpacaEval with win rate and MTBench with win score) and over-refusal (ORBench-Hard and XSTest with refusal rate) evaluations, to distinguish safety improvements from simply refusing more often. The results below reveal two observations: (i) UARM's stronger safety performance accompanies comparable or better helpfulness and over-refusal results relative to MCNF and Clear; (ii) all three uncertainty-aware methods improve helpfulness and reduce benign refusal relative to Naive. This supports safety gains without a broad usefulness trade-off relative to the evaluated baselines. We have added these results and analysis to Appendix D.8.
MethodAlpacaEval-WRAlpacaEval-RIMTBench-ScoreMTBench-RIORBench-Hard-RRORBench-Hard-RIXSTest-RRXSTest-RI
Naive5.87%-5.206-20.70%-1.60%-
MCNF10.43%77.78%↑\uparrow5.82511.88%↑\uparrow15.69%24.18%↑\uparrow0.40%75.00%↑\uparrow
Clear10.43%77.78%↑\uparrow5.83112.00%↑\uparrow15.39%25.64%↑\uparrow0.80%50.00%↑\uparrow
UARM10.65%81.48%↑\uparrow5.84412.24%↑\uparrow15.01%27.47%↑\uparrow0.80%50.00%↑\uparrow
  • Third, we clarify the choice of short training and one judge. We extend Qwen3-8B training to 60 updates and evaluate FFT and WildGuardMix every 10 updates. The results below and Figure 4 in the revised manuscript show that both safety scores peak at step 20 and decline afterward, while reward improvement slows and uncertainty generally decreases, supporting the 20-update checkpoint in this setting. We have added the curves and analysis to Appendix D.5. Besides, we use one judge to evaluate safety, following common LLM post-training practice [1,2], and this ensures a consistent evaluation criterion across methods and benchmarks.
Step102030405060
Reward-0.050-0.029-0.010-0.002-0.007-0.003
Uncertainty0.1150.0940.0700.0660.0690.049
FFT-SS0.5250.6050.5590.5130.4920.489
WildGuardMix-SS0.7070.7950.7590.7450.7190.710

[1] Miao et al. Inform: Mitigating reward hacking in rlhf via information-theoretic reward modeling. NeurIPS, 2024.

[2] Ye et al. Learning llm-as-a-judge for preference alignment. ICLR, 2025.

  • Finally, we add multi-seed GRPO experiments to address the single-run concern. With seeds 1, 2, and 42, the results below show that UARM achieves the highest mean safety score on all four benchmarks. Its variability is lower on HarmBench and DAN but higher on FFT and WildGuardMix than MCNF and Clear. The consistent mean advantage supports repeatability of the safety gains. We have added the seed sensitivity results and analysis to Appendix D.7.
MethodHarmBenchFFTDANWildGuardMix
Naive0.539$_{\pm 0.014}$0.425$_{\pm 0.021}$0.560$_{\pm 0.006}$0.645$_{\pm 0.003}$
MCNF0.610$_{\pm 0.015}$0.524$_{\pm 0.011}$0.612$_{\pm 0.005}$0.723$_{\pm 0.006}$
Clear0.574$_{\pm 0.008}$0.510$_{\pm 0.009}$0.603$_{\pm 0.012}$0.707$_{\pm 0.013}$
UARM0.661$_{\pm 0.006}$0.613±0.013_{\pm 0.013}0.689±0.005_{\pm 0.005}0.780_{\pm 0.016}\

[W3] Predicting quantiles and then calibrating the interval on a held-out set is close to existing methods such as CQR. Theorem 3.1 follows from the standard calibration-set argument, and the GRPO reweighting is essentially inverse-variance weighting. More importantly, the coverage guarantee assumes calibration data and test samples come from the same distribution, but during RLHF the policy keeps changing, so the guarantee does not hold exactly where the method is meant to be used.

Response. Thank you for your observant and thoughtful comment. Our clarifications are detailed as follows.

  • First, we clarify the distinction between UARM's interval construction and CQR. Specifically, CQR corrects a fixed pair of quantile endpoints, whereas UARM calibrates the number of interquantile intervals merged along a greedy nested path. UARM's dense quantile grid captures the predicted reward-distribution shape and supports asymmetric interval expansion, with its median quantile serving as the point reward. We have clarified the construction and its ideal shortest-block objective in Section 3.2.

  • Second, we clarify Theorem 3.1's relation to the standard calibration-set argument. The theorem uses the standard CP calibration principle, with nestedness required for our interval construction. This additional nestedness assumption is satisfied by our greedy construction, which retains the previous interval at each expansion so that an included reward remains covered as the interval grows. We have clarified this requirement in Section 3.2.

  • Third, we clarify the distinction between GRPO reweighting and inverse-variance weighting. Direct inverse-noise-variance weighting uses wi∝1/σnoise,i2w_i\propto 1/\sigma_{\mathrm{noise},i}^2. In contrast, UARM uses the bounded attenuation wi=σsignal2/(σsignal2+σnoise,i2)w_i=\sigma_{\mathrm{signal}}^2/(\sigma_{\mathrm{signal}}^2+\sigma_{\mathrm{noise},i}^2), combining interval-derived uncertainty with group-level signal variation. This signal-to-noise shrinkage is related to inverse-variance weighting but remains bounded as noise approaches zero. It scales (ri−μ)/σsignal(r_i-\mu)/\sigma_{\mathrm{signal}} within the GRPO surrogate without additional RM forward passes.

  • Finally, we agree that policy-induced distribution shift can limit the coverage guarantee. Theorem 3.1 assumes exchangeability conditional on the fitted model. Theorem 3.2 further requires independent i.i.d. splits and uniform quantile calibration. We have acknowledged this shortcoming in our Limitations section. Adaptive conformal prediction or semi-supervised reward-model retraining using retained high-confidence rollouts offers promising avenues for addressing such drift.

Reviewer_ioTDOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

Reward models return one unconfident score, so GRPO's group normalization lets a single unreliable extreme reward dominate the group mean. UARM makes the reward model a quantile regressor, calibrates a nested interval on held-out data, and maps interval width to a noise variance that attenuates GRPO advantages. Evaluated on three preference datasets, 12 UQ baselines, five backbones, and a downstream safety study.

Critical strengths
  1. The interface that converts uncertainty into an advantage weight is the useful part: per-sample uncertainty comes from an interval already computed, with no extra forward pass, and the construction reduces to standard GRPO when noise is uniform.

  2. Breadth is adequate: 12 baselines, three reward scales, five backbones, ablation, sensitivity. Table 4 shows three different uncertainty sources all improving safety under one integration, so the channel does carry signal.

Critical weaknesses
  1. R²@50 and MSE@50 use each method's own selected 50%, which changes both the error and the R² denominator, and better point accuracy mechanically puts more samples in the confident half. Table 1 therefore cannot separate better ranking from a better reward model.

  2. Gains cannot be attributed to uncertainty: §4.6 assumes a shared GRPO integration, but the reward models differ in objective and the MCNF/Clear intervals differ in construction and scale, so Eq. (12) yields non-commensurate attenuation; the gain could come from a better median reward.The setting is also narrow: safety only, one LLM judge, 20 update steps, 4 rollouts, no seed variance, and no over-refusal, helpfulness, length or diversity measurement. "Stronger downstream alignment" is therefore not established.

Questions for authors
  1. Which construction produced the numbers, Eq. (7)'s globally narrowest block or C.2's greedy expansion? They differ, and Thm 3.1's nestedness holds for the latter only.
  2. Please report empirical coverage, mean width and risk–coverage/AURC on all three test sets, and state whether the early-stopping validation set is disjoint from D_cal as B.2 requires.
  3. What is the reward sign convention for PKU-SafeRLHF, where severity rises with harm while GRPO maximizes reward?
  4. §4.6 states a 0.042 HarmBench gap on LLaMA3.1-8B, but Table 4 gives 0.755 − 0.709 = 0.046; please confirm which is correct.
6 comments

Thank you for your detailed feedback and appreciation of our methodology, practical utility, and empirical evaluation. Below are our responses to your specific concerns.


[W1,Q2] Table 1 cannot separate better ranking from a better reward model because each method uses its own selected 50%, changing both the error and the R$^2$ denominator. Please report empirical coverage, mean width and risk-coverage/AURC on all three test sets, and state whether the early-stopping validation set is disjoint from Dcal\mathcal D_\mathrm{cal} as B.2 requires.

Response. Thank you for your insightful and constructive suggestion. Our responses and actions are organized as follows.

  • First, we agree that @50 metrics alone only partially reflect the effectiveness of uncertainty estimates: they measure accuracy after uncertainty-based selection, but do not isolate better ranking from point-prediction quality or assess interval calibration.

  • Accordingly, we add full-test-set R$^2$, MSE, MAE, empirical coverage (denoted as Coverage), mean interval width (denoted as Efficiency), and AURC, comparing UARM with Naive, MCNF, and Clear in the table below. There are two observations: (i) UARM achieves the smallest interval width and lowest AURC on all three datasets, with coverage close to the nominal 90% target; (ii) its full-set prediction metrics are not uniformly best, whereas it achieves the strongest @50 results on samples identified as most certain. This contrast supports accurate reward estimation on low-uncertainty samples, rather than uniform superiority in full-set accuracy or a claim of better ranking alone. We have added the table and analysis to Appendix D.3.

DatasetMethodR$^2$MSEMAECoverageEfficiencyAURC
HelpSteerNaive0.3530.5960.588---
MCNF0.4260.5290.5590.9162.1910.496
Clear0.4250.5290.5730.8842.2450.512
UARM0.4110.5420.5170.9012.0640.465
UltraFeedbackNaive0.5501.5360.887---
MCNF0.6681.1340.7590.9473.8570.550
Clear0.6691.1280.7930.9063.0540.546
UARM0.6651.1410.7470.9052.9110.514
PKU-SafeRLHFNaive0.8110.2190.290---
MCNF0.8360.1890.2420.9591.0920.083
Clear0.8210.2070.2670.8981.1120.096
UARM0.8270.2000.1720.9020.5610.044
  • Second, we use training loss for hyperparameter selection and early stopping, without a separate validation set. We use disjoint training, calibration, and test sets, with Dcal\mathcal D_\mathrm{cal} reserved for post-hoc calibration. Neither calibration nor test labels are used to fit or select the model. The references to validation-based tuning and early stopping were reporting errors. We have corrected these descriptions in Section 4.1 and Appendix C.2.

[W2] Gains cannot be attributed to uncertainty: despite shared GRPO integration, different reward objectives and MCNF/Clear interval constructions and scales yield non-commensurate attenuation in Eq. (12); a better median reward could explain the gains. Stronger downstream alignment is not established by safety-only evaluation with one judge, 20 updates, four rollouts, no seed variance, and no over-refusal, helpfulness, length, or diversity measurements.

Response. Thank you for your meticulous and helpful suggestion. Our actions and responses are detailed as follows.

  • First, to distinguish the contributions of uncertainty reweighting and median-reward quality, we add paired GRPO experiments with identical rewards, with and without reweighting (†^\dagger). The results below show that reweighting improves safety on all four benchmarks. UARM's reward-only performance is comparable to the other controls, while full UARM performs best, supporting gains beyond a better median-reward estimate. We have added the results and analysis to Appendix D.4.
MethodHarmBench-SSHarmBench-RIFFT-SSFFT-RIDAN-SSDAN-RIWildGuardMix-SSWildGuardMix-RI
Naive0.540-0.445-0.555-0.642-
MCNF$^\dagger$0.5736.11%↑\uparrow0.4736.41%↑\uparrow0.5824.74%↑\uparrow0.6694.11%↑\uparrow
MCNF0.61313.52%↑\uparrow0.53219.64%↑\uparrow0.61811.28%↑\uparrow0.73013.70%↑\uparrow
Clear$^\dagger$0.5573.15%↑\uparrow0.4623.84%↑\uparrow0.5773.84%↑\uparrow0.6572.29%↑\uparrow
Clear0.5848.06%↑\uparrow0.50814.09%↑\uparrow0.6028.45%↑\uparrow0.71911.88%↑\uparrow
UARM$^\dagger$0.5715.74%↑\uparrow0.4715.98%↑\uparrow0.5743.46%↑\uparrow0.6775.48%↑\uparrow
UARM0.66423.06%↑\uparrow0.60535.90%↑\uparrow0.69525.13%↑\uparrow0.79523.75%↑\uparrow
  • Second, we clarify the choice of one judge and 20 updates. A shared LLM judge provides a consistent safety criterion across methods and benchmarks, following common LLM post-training practice [1,2]. To assess the update budget, we extend Qwen3-8B training to 60 updates and evaluate FFT and WildGuardMix every 10 updates. The results below and Figure 4 in our revised manuscript show that both safety scores peak at step 20 and subsequently decline, while reward improvement slows and uncertainty decreases. These observations support choosing 20 updates in this setting. We have added the optimization curves and analysis to Appendix D.5.
Step102030405060
Reward-0.050-0.029-0.010-0.002-0.007-0.003
Uncertainty0.1150.0940.0700.0660.0690.049
FFT-SS0.5250.6050.5590.5130.4920.489
WildGuardMix-SS0.7070.7950.7590.7450.7190.710

[1] Miao et al. Inform: Mitigating reward hacking in rlhf via information-theoretic reward modeling. NeurIPS, 2024.

[2] Ye et al. Learning llm-as-a-judge for preference alignment. ICLR, 2025.

  • Third, we add experiments varying the number of rollouts per prompt from 4 to 8, 12, and 16 to assess sensitivity to group size. The results below show that performance generally improves as Nrol\mathrm N_{\mathrm{rol}} increases, indicating that the gains are not confined to four rollouts. We have added the sensitivity results and analysis to Appendix D.6.
Nrol\mathrm{N}_\mathrm{rol}HarmBench-SSHarmBench-RIFFT-SSFFT-RIDAN-SSDAN-RIWildGuardMix-SSWildGuardMix-RI
Naive0.540-0.445-0.555-0.642-
40.66423.06%↑\uparrow0.60535.90%↑\uparrow0.69525.13%↑\uparrow0.79523.75%↑\uparrow
80.68426.76%↑\uparrow0.61237.60%↑\uparrow0.71328.33%↑\uparrow0.78321.92%↑\uparrow
120.70730.93%↑\uparrow0.62239.74%↑\uparrow0.72129.86%↑\uparrow0.79824.21%↑\uparrow
160.70931.30%↑\uparrow0.63141.87%↑\uparrow0.72630.76%↑\uparrow0.80625.58%↑\uparrow
  • Fourth, we add multi-seed GRPO experiments to assess seed variance. Using seeds 1, 2, and 42, the results below show that UARM achieves the highest mean safety score on all benchmarks, with variation generally smaller than MCNF and Clear. These results support repeatability of the safety gains. We have added the seed sensitivity results and analysis to Appendix D.7.
MethodHarmBenchFFTDANWildGuardMix
Naive0.539$_{\pm 0.014}$0.425$_{\pm 0.021}$0.560$_{\pm 0.006}$0.645$_{\pm 0.003}$
MCNF0.610$_{\pm 0.015}$0.524$_{\pm 0.011}$0.612$_{\pm 0.005}$0.723$_{\pm 0.006}$
Clear0.574$_{\pm 0.008}$0.510$_{\pm 0.009}$0.603$_{\pm 0.012}$0.707$_{\pm 0.013}$
UARM0.661$_{\pm 0.006}$0.613±0.013_{\pm 0.013}0.689±0.005_{\pm 0.005}0.780_{\pm 0.016}\
  • Finally, we add helpfulness and over-refusal experiments to complement the safety-only evaluation. The results below reveal two observations: (i) UARM's stronger safety performance is accompanied by comparable or better helpfulness and over-refusal results relative to MCNF and Clear; (ii) all three uncertainty-aware methods improve helpfulness and reduce over-refusal relative to Naive. These findings support safety gains without a broad usefulness trade-off in the evaluated setting. We have added the results and analysis to Appendix D.8.
MethodAlpacaEval-WRAlpacaEval-RIMTBench-ScoreMTBench-RIORBench-Hard-RRORBench-Hard-RIXSTest-RRXSTest-RI
Naive5.87%-5.206-20.70%-1.60%-
MCNF10.43%77.78%↑\uparrow5.82511.88%↑\uparrow15.69%24.18%↑\uparrow0.40%75.00%↑\uparrow
Clear10.43%77.78%↑\uparrow5.83112.00%↑\uparrow15.39%25.64%↑\uparrow0.80%50.00%↑\uparrow
UARM10.65%81.48%↑\uparrow5.84412.24%↑\uparrow15.01%27.47%↑\uparrow0.80%50.00%↑\uparrow

[Q1] Which construction produced the numbers, Eq. (7)'s globally narrowest block or C.2's greedy expansion? They differ, and Thm 3.1's nestedness holds for the latter only.

Response. Thank you for your meticulous and insightful comment. Our clarifications and actions are detailed as follows.

  • First, our interval construction is the greedy expansion described in Appendix C.2, while the globally narrowest block in Eq. (7) is the ideal width-minimization objective. We choose greedy expansion because independently selected global minima need not be nested. Nestedness ensures that once a reward is included, it remains covered at all larger interval counts, allowing the first-inclusion score to be inverted into a calibrated prediction interval. We have further clarified this distinction in Section 3.2, separating the ideal family Jm⋆\mathcal J_m^{\star} from the greedy family Jm\mathcal J_m used for calibration and prediction, and updated Eq. (9) accordingly.

  • Second, we agree that Theorem 3.1's nestedness holds for the greedy expansion only. Each expansion retains the previous interval, ensuring Jm(x)⊆Jm+1(x)\mathcal J_m(x)\subseteq\mathcal J_{m+1}(x). We have clarified in Section 3.2 and Assumption B.3 that the coverage guarantee applies to this nested family, rather than independently selected global minima.

[Q3] What is the reward sign convention for PKU-SafeRLHF, where severity rises with harm while GRPO maximizes reward?

Response. Thank you for your observant comment. Our reward sign convention is to flip the predicted severity before GRPO: rpolicy(x)=−q^K/2(x)r_{\mathrm{policy}}(x)=-\hat q_{\mathrm{K}/2}(x). Thus, maximizing reward favors lower harmfulness, while interval width remains unchanged. We have clarified this convention in Appendix C.2.

[Q4] §4.6 states a 0.042 HarmBench gap on LLaMA3.1-8B, but Table 4 gives 0.755 - 0.709 = 0.046; please confirm which is correct.

Response. Thank you for your meticulous comment. The 0.042 HarmBench gap was a transcription error, and the correct value is 0.046 (0.755−0.7090.755-0.709) as you noted. We have corrected it in Section 4.6.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Author Final Remarks by Authors

Dear AC and all reviewers,

We sincerely appreciate your commitment to evaluating our paper despite your busy schedules. We are encouraged that 2 out of 5 reviewers are currently on the positive side, with scores 3 with conf 2 and 3 with conf 3, recognizing the following strengths of our paper:

  • "The problem of reward models giving no sense of confidence is easy to understand, and the paper explains it well" (Reviewer 2djc),
  • "Theoretically honest coverage analysis" and "Comprehensive baselines, consistent gains, negligible overhead" (Reviewer MCAR).

At the same time, we noticed that Reviewers ioTD, krFA, and ZQk7 gave initial ratings of 2 with conf 2, 2 with conf 3, and 2 with conf 3, respectively (no discussion responses yet). After acknowledging in their initial reviews that "The interface that converts uncertainty into an advantage weight is the useful part" (Reviewer ioTD), "The paper identifies a genuine gap in the standard RLHF pipeline and proposes a practical solution" (Reviewer krFA), and "The experimental scope is broad" (Reviewer ZQk7), they raised concerns about (1) evaluation of uncertainty and calibration, (2) downstream attribution, usefulness, and experimental rigor, and (3) methodological distinctions and assumptions.

"Table 1 therefore cannot separate better ranking from a better reward model" (Reviewer ioTD); "report empirical coverage by uncertainty decile" (Reviewer krFA); "Coverage is unsupported and unmeasured" (Reviewer ZQk7).

  • We sincerely appreciate the request for more empirical evidence and have added full-test-set and five-run random-half metrics, risk-coverage and width-error curves, empirical coverage and mean width, and coverage by uncertainty decile (please kindly refer to our full response). Our method achieves the smallest mean interval width and lowest AURC with coverage close to the nominal 90% target across all datasets. Its uncertainty-selected half also outperforms its random half, supporting the benefit of uncertainty-based selection.

"Gains cannot be attributed to uncertainty" (Reviewer ioTD); "Without evidence that helpfulness is preserved, the safety improvements may come at an unreported cost" (Reviewer krFA); "Experiment 2 lacks controls" and "Safety cannot be told apart from refusal" (Reviewer ZQk7).

  • We sincerely appreciate these concerns and have added controls using UARM's reward with standard GRPO, an ensemble penalty, and shared-noise and shuffled-width variants, supporting the contribution of our per-rollout uncertainty weighting.
  • We have also added AlpacaEval, MT-Bench, ORBench-Hard, and XSTest evaluations, which support safety gains in our method without degraded helpfulness or increased over-refusal.

"Limited methodological novelty" (Reviewer krFA); "No derivation" and "The coverage guarantees require rollouts to be exchangeable with the calibration data" (Reviewer ZQk7).

  • We respectfully clarify that our method connects calibrated reward uncertainty to per-rollout GRPO updates. Our added comparisons show stronger uncertainty-selected prediction accuracy than CIR, CHR, and a categorical head across all datasets.
  • We also derive the attenuation factor as minimizing reward-estimation MSE under explicit assumptions, providing a statistical rationale for downweighting less reliable rewards.
  • We agree that on-policy coverage evaluation is important, but reward annotation was infeasible within the rebuttal's time and cost constraints. We explicitly retain the exchangeability requirement and discuss adaptive calibration and semi-supervised retraining as future directions.

We are confident that our responses and additional experiments can thoroughly address the reviewers' concerns. We kindly remind that Reviewers ioTD, krFA, ZQk7, and MCAR have not yet replied to our responses. Thus, we respectfully ask the AC and reviewers to consider this context when making your recommendation. Thank you for your invaluable time and effort.

Many thanks,

Authors of #54220

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.