acceptodds
Under review as a conference paper at ICLR 2027

Safe RLHF Beyond Expectation: Stochastic Dominance for Spectral Risk Control

Abstract

Safe Reinforcement Learning from Human Feedback (RLHF) typically enforces safety by constraining the expected cost of a policy. However, expected cost captures only average behavior and can overlook rare but severe failures. We propose Risk-sensitive Alignment via Dominance (RAD), a distributional approach to safe RLHF that compares the cost distribution of the learned policy to that of a reference policy rather than only their means. RAD represents policy-induced cost distributions non-parametrically through quantiles and encourages the learned policy to have lower cost quantiles than the reference policy. To optimize this distributional objective for language models, RAD uses a quantile-particle representation that connects two gradient estimators: entropic optimal transport provides gradients of the distributional objective with respect to quantile particles, while score-function quantile gradients propagate these updates to the policy. We further generalize RAD with quantile weighting, allowing different regions of the cost distribution to be emphasized. For admissible spectral weights, the resulting objective has a direct connection to Spectral Risk Measures, providing a principled way to target different notions of risk. Empirically, on BeaverTails, RAD improves harmlessness over SFT, and several variants also improve harmlessness over Safe RLHF while maintaining similar helpfulness. On held-out HarmBench prompts, several risk-sensitive variants also improve harmlessness over Safe RLHF.

Primary area:
reinforcement learning
Keywords:
Reinforcement Learning from Human Feedback, Safety, Alignment, Stochastic Dominance, Optimal Transport, Spectral Risk Measures

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_ujD6OpenReview ↗
Rating
1
Confidence
4
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

This paper proposes Risk-sensitive Alignment via Dominance (RAD), a distributional safe RLHF framework that replaces expected-cost constraints with a quantile-based stochastic dominance objective. The method combines entropic optimal transport with score-function quantile gradients to obtain an approximate policy update. It further introduces quantile weighting to accommodate spectral risk measures. Experiments on BeaverTails and HarmBench show improvements in harmlessness over SFT and, for several variants, Safe RLHF.

Critical strengths

1. Meaningful extension of safe RLHF beyond expected costs.

The paper addresses an important limitation of expectation-based safety constraints by incorporating distributional information. The connection to spectral risk measures provides a flexible framework for expressing different risk preferences.

2. Empirical evaluation on real-world alignment benchmarks.

The experiments demonstrate promising harmlessness improvements on BeaverTails, with additional evaluation on held-out HarmBench prompts. These results provide preliminary evidence of the practical applicability of distributional safety objectives.

Critical weaknesses

W1. Insufficient theoretical justification of the proposed policy update.

Theorem 1 relies on several approximations whose effects are not adequately characterized.

  • Density-factor omission: Eq. (27) omits the quantile-density factor 1/fθ(qi)1/f_\theta(q_i). Although this preserves the direction of each individual quantile gradient, it does not generally preserve the ascent direction of their weighted sum. The authors acknowledge this limitation, but provide no conditions under which the proposed update improves the original objective.

  • Entropic regularization: The proposed update uses gradients of an entropically regularized OT objective. However, no bound is provided for the resulting gradient approximation error. Furthermore, the weighted extension in Eq. (35) is not rigorously shown to approximate the gradient of the weighted FSD objective under entropic regularization.

  • Stochastic dominance: The constraint in Eq. (6) encourages quantile-wise cost reductions but does not enforce first-order stochastic dominance. Although the authors characterize reverse violations, the relationship between the proposed optimization procedure and actual dominance guarantees remains limited.

Consequently, the main theoretical contribution provides a heuristic policy-update rule without establishing sufficient guarantees regarding the original distributional objective or constraint satisfaction.

W2. Insufficient experimental validation of the central claims.

The experimental design does not adequately establish the advantages of the proposed framework over simpler risk-sensitive alternatives.

  • Missing direct tail-risk evaluation: The paper is motivated by controlling rare but severe failures. However, the main evaluations focus on safe-response proportions, pairwise harmlessness judgments, and weighted dominance differences. Although the latter provide information about spectral risk, direct comparisons of tail-risk measures, such as CVaR at multiple confidence levels, are missing.

  • Missing risk-sensitive baselines: The principal trained baseline is expectation-constrained Safe RLHF. Without direct CVaR-constrained or spectral-risk-constrained RLHF baselines, it is unclear whether the improvements arise from the proposed RAD framework or simply from incorporating risk-sensitive objectives.

  • Missing algorithmic ablations: RAD and Safe RLHF employ different policy optimization algorithms. Moreover, no ablation isolates the contribution of entropic OT or examines the effect of omitting the quantile-density factors.

These limitations make it difficult to determine whether the additional complexity of RAD is necessary or provides advantages over direct risk-sensitive policy optimization.

Questions for authors

Q1. Theoretical justification of Theorem 1.

Can the authors establish conditions under which the approximate RAD update is an ascent direction for the original FSD objective? In particular, how do entropic regularization and density-factor omission affect the gradient approximation error?

Q2. Weighted FSD gradient (Eq. 35).

Can the authors justify why multiplying the entropic OT particle gradients by w(αi)w(\alpha_i) yields a valid approximation to the gradient of the weighted FSD objective?

Q3. Experimental validation.

Can the authors provide direct tail-risk comparisons (e.g., CVaR at multiple confidence levels) against Safe RLHF and a direct CVaR-constrained RLHF baseline under comparable optimization settings? An ablation without entropic OT would also help establish the necessity of the proposed optimization framework.

4 comments
AuthorsOpenReview ↗

Response by Authors [1/2]

W1. Theory of the policy update.

Answer. Each point holds RAD to a standard prior work does not meet.

Density factor. The omission is disclosed in Sec. 3, citing Li et al.'s (2024) density-free quantile estimator, as the review acknowledges. IQN (Dabney et al., ICML 2018) obtains CVaR-, Wang- and power-distorted policies by reweighting approximate quantiles without an ascent guarantee. The review nevertheless demands a guarantee absent from these methods. The revision supplies one for RAD's regularized FSD term (App. E). The density-free update ascends when per-quantile gradients are pairwise non-negatively aligned, regardless of their densities, or when inverse-density variation is below their gradient coherence. Equal densities guarantee ascent.

Entropic regularization. Like OT-based dominance methods (Farajzadeh et al., 2025), RAD uses transport for particle gradients (App. C). Entropy gives a unique, soft plan and well-defined particle gradients, converging to unregularized OT as χ→0\chi \to 0 (Peyré & Cuturi, 2020, Prop. 4.1). At χ=0.01\chi = 0.01, cosine similarity with exact OT particle gradients across three seeds is at least 0.97 initially and above 0.9999 at the trained RAD-Uniform and RAD-CVaR checkpoints (App. E). In unregularized OT, the weighted derivative is exactly w(αi)w(\alpha_i) times the unweighted derivative (App. E).

Dominance. Exact FSD requires an inequality at every quantile, so prior stochastic-dominance methods also use surrogates (Melnyk et al., 2024; Farajzadeh et al., 2025). RAD's surrogate attains empirical dominance. Across 7,495 held-out prompts, no reverse violation is detected for any variant or seed (App. H.1). The update therefore has stated ascent conditions, Corollary 3 bounds spectral-risk reduction, and the held-out data show empirical dominance. We ask that RAD be evaluated by the standard applied to established methods in this literature.

W2. Experimental validation.

Answer. Tail risk. Appendix H.1 adds full quantile curves and held-out CVaR₀.₈/₀.₉/₀.₉₅. RAD-CVaR is lowest at each level, −2.54/−0.59/1.09 versus Safe RLHF's −0.24/1.85/3.47. Its 95th-percentile cost is below the safety threshold, −1.20 versus 1.36. It retains a tail advantage under an independently trained cost model (App. H.2).

Risk-sensitive baseline. By Proposition 2, RAD is a spectral-risk-constrained Safe RLHF, with RAD-CVaR implementing the CVaR case and Wang and power corresponding to their weightings when reverse violation is zero. We additionally train Safe RLHF with the Rockafellar–Uryasev cost transform at the same 0.9 tail level and margin for three seeds. Both impose the same CVaR constraint at zero reverse violation (Corollary 3). RAD-CVaR has lower held-out CVaR at every reported level, −2.54/−0.59/1.09 versus −1.56/0.42/2.05 (App. H.1).

Optimizer ablation. This is RAD-Uniform, which uses the same pipeline without tail weighting. It matches Safe RLHF on every held-out cost statistic, while RAD-CVaR differs from it only in weighting and substantially lowers tail risk (App. H.1). The improvement comes from the objective rather than the optimizer.

Entropic regularization. Entropic regularization gives a unique, soft transport plan and well-defined particle gradients and recovers unregularized OT as χ→0\chi \to 0. Training particle gradients align with exact OT gradients at cosine similarity at least 0.97 initially and above 0.9999 at the trained checkpoints (W1, App. E).

Density factors. The policy cost densities at each quantile are unknown and unavailable in closed form. RAD follows Li et al. (2024) in omitting them. Neither Li et al.'s estimator nor IQN is evaluated against a version with exact densities, because that version does not exist without first constructing a new density estimator. Appendix E instead proves when omission preserves ascent. The requested ablation demands a standard no method in this line meets.

Q1. Ascent and approximations. See W1. Appendix E proves two ascent conditions for the regularized FSD term. The entropic solution converges to unregularized OT as χ→0\chi \to 0 (Peyré & Cuturi, 2020, Prop. 4.1), with measured particle-gradient cosine similarities of at least 0.97 initially and above 0.9999 at the trained checkpoints.

Q2. Weighted gradient. See W1. For unregularized OT, the weighted particle derivative is exactly w(αi)w(\alpha_i) times the unweighted derivative. The entropic version uses the same smoothing approximation (App. E).

Q3. Tail comparisons, CVaR baseline and OT ablation. See W2. Appendix H.1 adds CVaR at three levels and the direct Rockafellar–Uryasev Safe RLHF baseline at the same tail level and margin. RAD-CVaR has lower cost at each level. Training gradients closely match exact OT particle gradients (W1, App. E).

AuthorsOpenReview ↗

Response by Authors [2/2]

Closing Statement: We consider the ratings unjustified and highly miscalibrated. The review assigns overall 1 despite soundness 2, defined as minor or fixable concerns. Fixable concerns cannot support clear rejection, and every stated concern is fixed in this revision. It assigns presentation 1, meaning difficult to follow despite careful reading, without identifying an unclear passage, while other reviewers rate presentation 3 and 2. Its own summary credits RAD with harmlessness gains over SFT and, for several variants, over Safe RLHF. It demands guarantees absent from established methods, treats the disclosed and cited Li et al. density-free estimator as a flaw, and requests an ablation against unknown densities. No method error is identified. A clear rejection is untenable, and we request that the reviewer reconsider the ratings accordingly.

Thank you for the additional results. The CVaR baseline and tail-risk evaluation address several of my concerns. However, it is still unclear whether the ascent conditions hold during training, and the weighted entropic update needs further justification. The benefit of OT over a simpler update also remains unclear. I think these points require substantial revision and additional experiments. Therefore, I will keep my original score.

AuthorsOpenReview ↗

Authors Response

Q1. Do the ascent conditions hold during training?

  • The objective RAD's update optimizes falls consistently in training. For RAD-CVaR, the weighted FSD violation falls from 8.73, 8.55 and 8.28 at the start of training to 0.52, 0.59 and 0.35 at the end in the three seeds, with most of the decrease in the first 200 to 400 steps.
  • The dual update then holds it at the constraint boundary. The update therefore consistently reduces the objective in practice, consistent with the ascent analysis in Appendix E.

Q2. Why the weighted entropic update?

  • Unregularized OT gives a hard, nonsmooth particle update. RAD's hinge cost is not strictly convex, so Theorem 4 does not guarantee a unique optimal plan, and the particle subgradients change abruptly at quantile crossings (Appendix D).
  • Entropic OT gives the update RAD uses. Its strictly convex penalty yields a unique soft transport plan, with particle gradients computed through Sinkhorn iterations (Appendix C).

Q3. What is the benefit of optimal transport over a simpler update?

  • The reviewer does not specify which simpler update. The natural alternative without optimal transport is a direct CVaR constraint, which we trained with the Rockafellar–Uryasev objective and REINFORCE/RLOO in RAD's own pipeline (RLOO-CVaR), alongside Safe RLHF's expected-cost objective in the same setup (Safe-RLOO).
MethodCVaR₀.₉CVaR₀.₉₅VaR₀.₉₅Safe (%)
Safe-RLOO1.60 ± 1.103.66 ± 1.381.04 ± 1.2093.5 ± 1.6
RLOO-CVaR0.71 ± 5.102.21 ± 5.140.14 ± 5.2691.1 ± 7.5
RAD-CVaR−0.59 ± 0.301.11 ± 0.43−1.20 ± 0.1797.0 ± 0.4
  • RAD-CVaR has the best tail metrics and safe rate in the same pipeline. It also has far smaller seed variation than the direct CVaR constraint (CVaR₀.₉ ±0.30 against ±5.10), consistent with the regularizing effect of its entropic transport coupling.
  • RAD also outperforms the CVaR-constrained Safe RLHF baseline at all three reported CVaR levels (Table 7).

On the score. The reviewer concedes that our CVaR baseline and tail-risk evaluation resolve several of their concerns. What remains are requests for further justification and unspecified additional experiments, and we have now run the comparison the review implies, against an update without optimal transport. None identifies an error in the method, and the reviewer rates the paper's soundness 2. Holding a clear rejection on these grounds contradicts the reviewer's own assessment, and would require rejecting any paper that leaves room for further justification. The reviewer acknowledges that several of their concerns were addressed, yet maintains the lowest possible rating. The rating is inconsistent with the review's own content, which undermines its reliability as an assessment of this paper. We ask the AC to determine, from the content of the review itself, whether the rating and the review retain any credibility.

Reviewer_AMZuOpenReview ↗
Rating
2(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2(+1)
Read review
Paper summary

This paper proposes a new framework for Safe RLHF by replacing the standard expected-cost constraint (CMDP) with a relaxed First-Order Stochastic Dominance (FSD) constraint comparing the policy’s cost distribution to that of a reference policy. The authors derive an approximate update direction for this objective and weight the different quantiles of the cost distribution using choices inspired by the family of spectral risk measures. They use Qwen2.5 3B for their empirical test and compare their results with SFT and Safe RLHF baselines on the BeaverTails and HarmBench benchmarks.

Critical strengths
  • Applying stochastic-dominance ideas to policy-induced safety-cost distributions in RLHF is interesting and addresses an important limitation of expected-cost constraints.
  • The connection between weighted quantile improvements and spectral risk measures provides a useful framework for expressing different risk preferences. Supporting multiple risk profiles through changes to the weighting function is practically appealing.
Critical weaknesses
  • While the authors test different weight functions, the experiments are limited to a single model (Qwen2.5-3B) and a single safety baseline (Safe RLHF).

  • The improvements over SFT in both helpfulness and harmlessness do not isolate the benefit of RAD, since SFT does not see the BeaverTails preference data during training, whereas RAD and Safe RLHF do. Standard RLHF would be a more informative additional baseline.

  • Safe RLHF could also be extended to optimize tail-sensitive cost constraints using the same weighting functions as RAD, by replacing the expected-cost constraint with a spectral-risk constraint. This would help distinguish the benefit of tail weighting from that of RAD’s particular formulation. Under uniform weighting, Safe RLHF actually achieves slightly better performance than RAD in Tables 2 and 5, although the differences are small relative to the reported variability.

  • Chaudhary et al. is an important missing paper that should be included as a baseline, yet it is not cited. It studies CVaR-based RLHF and is directly relevant to the paper’s motivation. Wachi et al. and Peng et al. are also relevant missing references and potential baselines:

    • Chaudhary et al., Risk-Averse Fine-tuning of Large Language Models, NeurIPS 2024.
    • Wachi et al., Stepwise Alignment for Constrained Language Model Policy Optimization, NeurIPS 2024.
    • Peng et al., Enhancing Safety in Reinforcement Learning with Human Feedback via Rectified Policy Optimization, NeurIPS 2025.
  • In Melnyk et al.’s hinge-based formulation, favorable quantile levels contribute zero, and minimization targets violating levels. RAD instead lower-bounds $ L_{\mathrm{FSD}}(C_{\pi_\theta},C_{\pi_{\mathrm{ref}}}) =\int_0^1[Q_{\mathrm{ref}}(q)-Q_\theta(q)]_+,dq. $ This term is zero at quantile levels where the learned policy’s cost exceeds the reference cost, and positive only where it improves on the reference. The constraint therefore rewards improvements while providing no direct penalty for regressions at violating quantiles. Could the authors explain why they chose to lower-bound this improvement term rather than minimize or upper-bound the reverse violation?

Questions for authors

What motivated the choice of GPT-4o-mini as the harmlessness judge, given the availability of newer, more capable models? Have the authors assessed the reliability of its judgments, for example, through agreement with human annotators or another judge, and whether the reported model rankings remain consistent across evaluators?

4 comments
AuthorsOpenReview ↗

Response by Authors

W1. One model and one baseline.

Answer. RAD improves LLM safety through a stochastic-dominance constraint relative to a reference policy within Safe RLHF. Its lower-bounded FSD objective connects to spectral-risk reduction, and one optimization framework handles any risk profile. We test seven weightings from expected cost to tail risk, each with three seeds. Safe RLHF is the canonical expected-cost constrained baseline, against which RAD produces empirically dominant policies with better tail-risk control. We also compare with direct CVaR-constrained Safe RLHF (W3). RAD-Uniform matches RAD's optimizer, sampling and compute, isolating weighting (Sec. 4). Another base model would test generality beyond Qwen2.5-3B, a scope question rather than a soundness flaw.

W2. SFT and standard RLHF.

Answer. SFT is the dominance reference. Safe RLHF sees the same BeaverTails preference data, and RAD-Uniform shares RAD's pipeline, making them the comparisons that isolate RAD. Standard RLHF has no cost model or safety constraint and is a weaker safety baseline. Dai et al. (2024, Sec. 4.2.2, Fig. 6a) already show Safe RLHF outperforming it on harmlessness. Adding standard RLHF would not further isolate RAD's objective.

W3. Spectral-risk Safe RLHF and uniform weighting.

Answer. A spectral-risk-constrained Safe RLHF is precisely what RAD is. Proposition 2 shows that, at zero reverse violation, RAD's weighted constraint equals the corresponding spectral-risk reduction relative to the reference. Thus RAD-CVaR is a CVaR-constrained Safe RLHF, and likewise for Wang and power weightings. We also trained the direct CVaR alternative in Safe RLHF's pipeline, applying its constraint to the Rockafellar–Uryasev transformed cost. The VaR is each batch's empirical 0.9-quantile. It uses the same tail level and margin below SFT as RAD-CVaR, with three seeds. At zero reverse violation, both impose the same CVaR constraint (Corollary 3). On 7,495 held-out prompts, RAD-CVaR has lower CVaR₀.₈/₀.₉/₀.₉₅ (−2.54/−0.59/1.09 vs. −1.56/0.42/2.05) and lower mean cost (−8.75 vs. −8.16) (App. H.1). Under uniform weighting, Safe RLHF's small advantage is within the reported variability, as expected when RAD-Uniform reduces to an expected-cost constraint.

W4. Missing references and potential baselines.

Answer. All three are discussed in App. G. RA-RLHF (Chaudhary et al., 2024) optimizes CVaR of one reward in unconstrained RLHF, with no cost model or safety constraint, and cannot be applied as proposed to our constrained problem. We test its CVaR principle on cost through the direct CVaR Safe RLHF baseline in W3, which RAD-CVaR outperforms at every reported level (App. H.1). SACPO (Wachi et al., 2024) uses stepwise DPO for the same expected-cost criterion as Safe RLHF, testing an optimizer rather than a different criterion. Safe RLHF and RAD-Uniform already cover expected cost. RePO (Peng et al., 2025) uses expected positive cost to require safety for every prompt–response pair, a fixed-threshold rather than distribution-comparison criterion. Our baselines therefore include canonical expected-cost Safe RLHF, optimizer-matched RAD-Uniform and directly CVaR-constrained Safe RLHF.

W5. Why lower-bound improvement?

Answer. This is deliberate (Sec. 3). Melnyk et al.'s hinge penalizes only quantiles worse than the reference and stops encouraging improvement once the reference is matched. RAD requires an improvement margin, continuing to lower cost until it is reached. Regressions are not ignored. Corollary 3 shows that reverse violation reduces spectral-risk improvement, and none is detected for any variant on held-out prompts (App. H.1).

Q. Why GPT-4o-mini, and is it reliable?

Answer. Its low cost permits thousands of pairwise comparisons across variants and seeds. We independently re-judge HarmBench with WildGuard (App. H.4), which finds every RAD variant harmful less often than Safe RLHF. On the same comparisons, the judges agree on 82.7% of cases when both give decisive verdicts (κ = 0.65), and their variant win-rate rankings are identical.

Closing Statement: We consider the overall rating of 1 and soundness rating of 1 unjustified and highly miscalibrated. Soundness 1 denotes significant concerns that do not appear easy to address, yet the review identifies no error in a derivation, proposition or proof. Its concerns about baselines and scope are addressed in a single revision. Nothing stated supports clear rejection or the lowest soundness rating, making the ratings untenable. Every concrete request is addressed, and we request that the reviewer reconsider the ratings accordingly.

Thank you for the detailed response and for all the work you put into the revision. The new Safe RLHF-CVaR baseline, the held-out tail analysis, and the added references are helpful. That said, while RAD-CVaR’s advantage over Safe RLHF-CVaR is clear under the training cost model (Table 7), the two are within one standard deviation of each other under the independent cost model (Table 10), so I’m not fully convinced the extra complexity of RAD is needed. Most RAD variants in Table 7 also have much larger standard deviations than Safe RLHF and Safe RLHF-CVaR, and only RAD-CVaR clearly improves on Safe RLHF-CVaR. Since RAD uses REINFORCE/RLOO while both Safe RLHF baselines use Safe RLHF’s critic-based pipeline, it’s hard to tell whether these differences in performance and variance come from the RAD formulation or just from the training pipeline. I also find the connection to spectral risk measures interesting, but the paper doesn’t give much guidance on when or why one weighting function should be preferred over another.

On W5, I see that reverse violations are zero or negligible in your experiments, but I still think the paper needs to explain why RAD is better than just optimizing a risk measure of the cost directly. With the quantile-particle representation, you could optimize the spectral risk of the cost with the same density-free estimator, for any weighting, and without optimal transport. This is also where my concern about the framing comes in. The title and introduction promise stochastic dominance and spectral-risk control, but the constraint only rewards cost reductions at quantiles where the policy already beats the reference and ignores the ones where it does worse. So both claims depend on the reverse violation being zero or small, which you observe empirically but the method doesn’t guarantee. The revision already calls the objective a relaxation in Section 3, but I think it would be more accurate to present it consistently as a dominance-inspired surrogate, including in the title, and to compare it against directly constraining the spectral risk of the cost. Given the pipeline issue above, that comparison would be most useful within RAD’s own setup, with the same REINFORCE/RLOO training and quantile particles and only the dominance term swapped out.

Despite these remaining concerns, I’ve raised my score to weak reject.

AuthorsOpenReview ↗

Authors Response [1/2]

We thank the reviewer for engaging and answer their concerns below:

Q1. RAD-CVaR and Safe RLHF-CVaR are within one standard deviation under the independent cost model (Table 10), so is RAD's extra complexity needed?

  • RAD is less complex in practice. It trains critic-free with REINFORCE/RLOO, and Sinkhorn adds minimal overhead. A full RAD run takes 8 hours on 2 A100s (16 GPU-hours), versus 4.5 days on 4 A100s (432 GPU-hours) for the official Safe RLHF implementation, roughly 27 times less compute.
  • Comparable performance is expected, and RAD-CVaR is still lower in every comparison. At zero reverse violation, both methods impose the same CVaR constraint (Corollary 3). RAD-CVaR has lower cost at every reported level under the independent cost model, within seed variation (Table 10), lower CVaR at every reported level under the training cost model (Table 7), and a higher safe rate at every threshold (Table 11).
  • RAD generalizes beyond CVaR. Changing only the weighting lets one formulation target any spectral risk. When the policy dominates the reference, every spectral risk is no greater than the reference's, and the selected risk improves by at least the margin (Corollary 3). Safe RLHF-CVaR constrains only CVaR.

Q2. Do the performance and variance differences come from RAD's formulation or its pipeline?

We trained Safe RLHF's expected-cost and CVaR objectives with REINFORCE/RLOO in RAD's setup (Safe-RLOO and RLOO-CVaR).

MethodPipelineCVaR₀.₉CVaR₀.₉₅VaR₀.₉₅Safe (%)
Safe RLHFCritic (official code)1.85 ± 0.073.47 ± 0.081.36 ± 0.1692.1 ± 0.1
Safe-RLOORLOO (RAD's setup)1.60 ± 1.103.66 ± 1.381.04 ± 1.2093.5 ± 1.6
RLOO-CVaRRLOO (RAD's setup)0.71 ± 5.102.21 ± 5.140.14 ± 5.2691.1 ± 7.5
RAD-CVaRRLOO (RAD's setup)−0.59 ± 0.301.11 ± 0.43−1.20 ± 0.1797.0 ± 0.4
  • The pipeline does not change the outcome. Safe RLHF and Safe-RLOO optimize expected cost through different pipelines and reach similar tail costs.
  • The objective does. In the same pipeline, RAD-CVaR improves every reported tail metric and the safe rate over Safe-RLOO. Its tail-weighted dominance difference is +36.83 ± 12.57 over Safe-RLOO and +50.32 ± 2.87 over Safe RLHF.
  • Direct CVaR is less stable in the same pipeline. RLOO-CVaR's CVaR₀.₉ varies ±5.10 across seeds, versus ±0.30 for RAD-CVaR, consistent with a regularizing effect from RAD's entropic transport coupling.
  • The variance pattern is consistent with quantile weighting. RAD-Uniform uses every quantile particle and varies across seeds comparably to Safe-RLOO. RAD-CVaR uses only the 0.9 and 0.95 particles and has the smallest seed variation among the RLOO-trained methods.

Q3. When should one weighting function be preferred?

The weighting encodes the risk requirement [1].

  • Worst outcomes matter (CVaR). CVaR accounts for the magnitude of tail losses, which VaR ignores, and it replaced VaR in the Basel market-risk capital framework [2]. RAD-CVaR applies this principle to the most harmful responses.
  • Harm should be penalized increasingly with severity (smooth spectra). Exponential spectra use a risk-aversion coefficient, as in futures clearinghouse margins [3]. Wang and proportional-hazard distortions serve related roles in insurance pricing [4, 5].
  • Expected harm suffices (uniform). Uniform weighting recovers Safe RLHF's expected-cost criterion.
  • RAD supports each choice by changing only the weighting. When the policy dominates the reference, every corresponding spectral risk is no greater than the reference's.

[1] C. Acerbi. Spectral measures of risk: A coherent representation of subjective risk aversion. Journal of Banking & Finance, 26(7), 2002.

[2] Basel Committee on Banking Supervision. Minimum capital requirements for market risk. Bank for International Settlements, 2016.

[3] J. Cotter and K. Dowd. Extreme spectral risk measures: An application to futures clearinghouse margin requirements. Journal of Banking & Finance, 30(12), 2006.

[4] S. Wang. Insurance pricing and increased limits ratemaking by proportional hazards transforms. Insurance: Mathematics and Economics, 17(1), 1995.

[5] S. Wang. A class of distortion operators for pricing financial and insurance risks. Journal of Risk and Insurance, 67(1), 2000.

AuthorsOpenReview ↗

Authors Response [2/2]

Q4. Why is RAD better than directly optimizing a spectral risk with the same particles and estimator?

  • Dominance controls every spectral risk, while a direct constraint controls only its selected risk. RAD optimizes for dominance over the reference, attained by every variant and seed in Table 9. When dominance holds, every spectral risk is no worse than the reference's. Without full dominance, the selected risk still improves whenever the margin exceeds the reverse violation (Corollary 3). A direct constraint does not compare the distributions quantile by quantile, so other risks can worsen while it is met. At zero reverse violation, the constraints coincide for the selected weighting, and Table 9 separately verifies empirical dominance for RAD.
  • Direct CVaR is far less stable in RAD's pipeline. RLOO-CVaR uses the Rockafellar–Uryasev objective. Against RAD-CVaR, it has CVaR₀.₉ of 0.71 ± 5.10 versus −0.59 ± 0.30, safe rate of 91.1 ± 7.5% versus 97.0 ± 0.4%, and reward of 2.52 ± 0.63 versus 3.37 ± 0.08.
  • The update structures differ. Rockafellar–Uryasev uses a sample-level hinge at estimated VaR, while RAD forms soft transport couplings before propagating particle updates. RAD-CVaR's greater stability is consistent with this difference.

Q5. Should the paper instead be framed as a dominance-inspired surrogate?

  • The paper already states the scope of its claim. The abstract says RAD encourages lower cost quantiles, and Sec. 3 presents the objective as a relaxation.
  • The mechanism is realized in practice. No reverse violation is detected for any variant or seed (Table 9), so every evaluated run attains empirical dominance and the associated spectral-risk control.

The matched-pipeline comparisons show RAD-CVaR's stronger tail results and smaller seed variation, while the official-code comparison shows its substantially lower compute cost. We ask the reviewer to reconsider their score in light of these results.

Reviewer_FtLaOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies safe RLHF from a distributional perspective. Instead of constraining only the expected cost, it proposes RAD, which compares the cost quantiles of the learned policy with those of a reference policy. The cost distributions are represented using empirical quantile particles, and the training update combines an entropic optimal-transport gradient with a score-function estimator for the quantiles. The method also allows different quantile weightings, so that training can focus on either average cost or higher-cost parts of the distribution; the paper relates these choices to spectral risk measures. Experiments with Qwen2.5-3B on BeaverTails show improvements over SFT and mixed but sometimes favorable results against Safe RLHF. Several variants also outperform Safe RLHF in a pairwise harmlessness evaluation on held-out HarmBench prompts.

Critical strengths
  1. The paper focuses on a real weakness of expected-cost constraints. A policy can have a low average cost while still producing a small number of seriously harmful responses. Looking at cost quantiles is therefore a sensible way to study behavior that would be missed by the mean, especially in safety-critical settings.

  2. The proposed framework is flexible and shows encouraging initial results. RAD can place different amounts of weight on different parts of the cost distribution without requiring a separate training method for each risk measure. The HarmBench experiment is also useful: it evaluates the models on prompts that were not used to train the policy or the cost model, and several RAD variants perform better than Safe RLHF in the pairwise harmlessness comparison.

Critical weaknesses
  1. The proposed objective does not actually enforce stochastic dominance. It rewards quantiles at which the learned policy already has lower cost than the reference, but does not penalize quantiles at which it has higher cost. The small reverse violations in Table 3 are encouraging, but they are measured after training and are not guaranteed by the objective. This is important because stochastic-dominance control is the main claim of the paper, while the method is more accurately an FSD-inspired relaxation.

  2. The experiments do not clearly isolate the effect of the proposed objective. RAD uses REINFORCE/RLOO, while the Safe RLHF baseline uses a different critic-based training pipeline. A matched expected-cost baseline using the same optimizer and sampling setup is missing. In addition, most of the distributional results are evaluated with the same cost model used during training. HarmBench provides some independent evidence, but it only evaluates pairwise harmlessness and does not verify the claimed improvements in stochastic dominance or tail risk. These issues make it difficult to tell whether the gains come from the RAD objective itself.

Questions for authors
  1. Could the authors add an expected-cost baseline using the same REINFORCE/RLOO implementation as RAD? Ideally, this baseline would use the same number of samples per prompt, training steps, KL coefficient, and compute budget, so that the only difference is the safety objective.

  2. Could the authors show the actual cost quantile curves for RAD, Safe RLHF, and SFT on held-out data? Reporting high quantiles and CVaR values, together with reverse violations, would make it easier to see whether RAD really improves the tail or whether the aggregate metrics are hiding crossings between the distributions.

  3. How are responses converted from continuous cost-model scores into the binary safe/unsafe labels reported in Table 2? It would be helpful to specify the threshold, how it was selected and calibrated, and the sensitivity of the reported safe-response proportions to this threshold.

3 comments
AuthorsOpenReview ↗

Response by Authors

W1. Does the objective enforce stochastic dominance?

Answer. RAD optimizes an FSD relaxation (Sec. 3). Exact FSD requires an inequality at every quantile, which cannot be imposed directly in gradient-based training, so prior work likewise uses OT surrogates (Melnyk et al., 2024; Farajzadeh et al., 2025). Melnyk et al.'s hinge gives no incentive to improve once the policy matches the reference. RAD instead requires a margin of improvement. Corollary 3 accounts for regressions through the reverse violation, which reduces spectral-risk improvement. On 7,495 held-out prompts, reverse violation is zero for every variant and seed (App. H.1), and at most 0.02 under an independent cost model (App. H.2).

W2. Do the experiments isolate RAD's objective?

Answer. RAD-Uniform is the matched expected-cost control (Sec. 4). It uses the same REINFORCE/RLOO optimizer, responses per prompt, generation settings, training steps, KL coefficient and compute as the other RAD variants. Its constraint reduces to expected cost once dominance holds. It matches Safe RLHF on every held-out cost statistic, while RAD-CVaR differs only in weighting and substantially improves the tail (App. H.1). The gain comes from the objective, not the optimizer.

Tables 3–4 use the training cost model. The revision adds an independently trained Beaver-7b model, whose scores correlate with ours at Spearman 0.43. Every RAD variant nearly dominates the reference under this model (reverse violation at most 0.02), and RAD-CVaR retains its tail advantage over Safe RLHF (95th-percentile cost −0.64 vs. −0.02) (App. H.2). Appendix H.1 adds full held-out quantile curves and CVaR at three levels under the training cost model. A second independent HarmBench judge, WildGuard, confirms the pairwise results (App. H.4).

Q1. Matched expected-cost baseline. This is RAD-Uniform, which shares RAD's implementation, responses per prompt, training steps, KL coefficient and compute budget. Only the safety objective differs (W2).

Q2. Quantiles, CVaR and reverse violations. Appendix H.1 shows lower cost than SFT at every quantile for every RAD variant. The curves against Safe RLHF cross near the median, but RAD-CVaR is lower from the 75th percentile upward, so the aggregate metrics do not hide an adverse tail crossing. RAD-CVaR has the lowest CVaR at all three levels, with CVaR₀.₈/₀.₉/₀.₉₅ of −2.54/−0.59/1.09 versus Safe RLHF's −0.24/1.85/3.47. Its 95th-percentile cost is below the safety threshold (−1.20 vs. 1.36). No reverse violation against SFT is detected for any variant.

Q3. Binary safety labels. Following Safe RLHF, cost at most 0 denotes a safe response. The threshold is not tuned (Sec. 4). At the stricter −1 and looser +1 thresholds, RAD-CVaR has the highest safe rate, 95.5% and 97.9%, versus 89.4% and 94.3% for Safe RLHF (App. H.3).

Closing Statement: We thank the reviewer for the specific feedback, which shaped the matched-baseline framing, held-out quantile analysis and threshold sensitivity analysis. We request reassessment of the paper and score in light of these additions.

As the discussion period closes today, we would be grateful if the reviewer could take a look at our response and the revised paper. We are happy to answer any remaining questions.

Thank you for the response. I also considered the additional Safe-RLOO and RLOO-CVaR comparisons provided in the other review threads. These matched-pipeline comparisons, together with the held-out tail-risk evaluation and independent evaluators, substantially address my concerns about attribution and evaluation. The objective still does not guarantee stochastic dominance, and this limitation should remain explicit. Overall, I find the contribution useful and support raising my rating to 3.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Common Response

The revision addresses the main shared concerns, with all changes marked in blue.

  1. Held-out evaluation (App. H). On 7,495 held-out prompts, we report CVaR at three levels and full quantile curves (H.1), reverse violations, re-scoring with an independently trained cost model (H.2), sensitivity to the safety threshold (H.3), and a second independent HarmBench judge, WildGuard (H.4).

  2. CVaR-constrained Safe RLHF baseline (App. H.1). We train Safe RLHF with a Rockafellar–Uryasev CVaR constraint at the same tail level and margin as RAD-CVaR. RAD-CVaR has lower cost at every CVaR level.

  3. Theory of the policy update (App. E). We prove two conditions under which RAD's density-free direction is an ascent direction for the regularized FSD term, show that the weighted gradient is exact in the unregularized form, and report that the particle gradients used in training are closely aligned with exact OT gradients.

  4. Framing and setup (Secs. 3–4). The objective is presented as a relaxation of FSD, RAD-Uniform is identified as the matched expected-cost baseline, and the safety threshold is stated explicitly.

  5. Related work (App. G). RA-RLHF, SACPO, RePO and IQN are now discussed.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.