- Rating
- 1
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 1
- Soundness
- 2
Read review
Paper summary
This paper proposes Risk-sensitive Alignment via Dominance (RAD), a distributional safe RLHF framework that replaces expected-cost constraints with a quantile-based stochastic dominance objective. The method combines entropic optimal transport with score-function quantile gradients to obtain an approximate policy update. It further introduces quantile weighting to accommodate spectral risk measures. Experiments on BeaverTails and HarmBench show improvements in harmlessness over SFT and, for several variants, Safe RLHF.
Critical strengths
1. Meaningful extension of safe RLHF beyond expected costs.
The paper addresses an important limitation of expectation-based safety constraints by incorporating distributional information. The connection to spectral risk measures provides a flexible framework for expressing different risk preferences.
2. Empirical evaluation on real-world alignment benchmarks.
The experiments demonstrate promising harmlessness improvements on BeaverTails, with additional evaluation on held-out HarmBench prompts. These results provide preliminary evidence of the practical applicability of distributional safety objectives.
Critical weaknesses
W1. Insufficient theoretical justification of the proposed policy update.
Theorem 1 relies on several approximations whose effects are not adequately characterized.
-
Density-factor omission: Eq. (27) omits the quantile-density factor . Although this preserves the direction of each individual quantile gradient, it does not generally preserve the ascent direction of their weighted sum. The authors acknowledge this limitation, but provide no conditions under which the proposed update improves the original objective.
-
Entropic regularization: The proposed update uses gradients of an entropically regularized OT objective. However, no bound is provided for the resulting gradient approximation error. Furthermore, the weighted extension in Eq. (35) is not rigorously shown to approximate the gradient of the weighted FSD objective under entropic regularization.
-
Stochastic dominance: The constraint in Eq. (6) encourages quantile-wise cost reductions but does not enforce first-order stochastic dominance. Although the authors characterize reverse violations, the relationship between the proposed optimization procedure and actual dominance guarantees remains limited.
Consequently, the main theoretical contribution provides a heuristic policy-update rule without establishing sufficient guarantees regarding the original distributional objective or constraint satisfaction.
W2. Insufficient experimental validation of the central claims.
The experimental design does not adequately establish the advantages of the proposed framework over simpler risk-sensitive alternatives.
-
Missing direct tail-risk evaluation: The paper is motivated by controlling rare but severe failures. However, the main evaluations focus on safe-response proportions, pairwise harmlessness judgments, and weighted dominance differences. Although the latter provide information about spectral risk, direct comparisons of tail-risk measures, such as CVaR at multiple confidence levels, are missing.
-
Missing risk-sensitive baselines: The principal trained baseline is expectation-constrained Safe RLHF. Without direct CVaR-constrained or spectral-risk-constrained RLHF baselines, it is unclear whether the improvements arise from the proposed RAD framework or simply from incorporating risk-sensitive objectives.
-
Missing algorithmic ablations: RAD and Safe RLHF employ different policy optimization algorithms. Moreover, no ablation isolates the contribution of entropic OT or examines the effect of omitting the quantile-density factors.
These limitations make it difficult to determine whether the additional complexity of RAD is necessary or provides advantages over direct risk-sensitive policy optimization.
Questions for authors
Q1. Theoretical justification of Theorem 1.
Can the authors establish conditions under which the approximate RAD update is an ascent direction for the original FSD objective? In particular, how do entropic regularization and density-factor omission affect the gradient approximation error?
Q2. Weighted FSD gradient (Eq. 35).
Can the authors justify why multiplying the entropic OT particle gradients by yields a valid approximation to the gradient of the weighted FSD objective?
Q3. Experimental validation.
Can the authors provide direct tail-risk comparisons (e.g., CVaR at multiple confidence levels) against Safe RLHF and a direct CVaR-constrained RLHF baseline under comparable optimization settings? An ablation without entropic OT would also help establish the necessity of the proposed optimization framework.
4 comments
Response by Authors [2/2]
Closing Statement: We consider the ratings unjustified and highly miscalibrated. The review assigns overall 1 despite soundness 2, defined as minor or fixable concerns. Fixable concerns cannot support clear rejection, and every stated concern is fixed in this revision. It assigns presentation 1, meaning difficult to follow despite careful reading, without identifying an unclear passage, while other reviewers rate presentation 3 and 2. Its own summary credits RAD with harmlessness gains over SFT and, for several variants, over Safe RLHF. It demands guarantees absent from established methods, treats the disclosed and cited Li et al. density-free estimator as a flaw, and requests an ablation against unknown densities. No method error is identified. A clear rejection is untenable, and we request that the reviewer reconsider the ratings accordingly.
Thank you for the additional results. The CVaR baseline and tail-risk evaluation address several of my concerns. However, it is still unclear whether the ascent conditions hold during training, and the weighted entropic update needs further justification. The benefit of OT over a simpler update also remains unclear. I think these points require substantial revision and additional experiments. Therefore, I will keep my original score.
Authors Response
Q1. Do the ascent conditions hold during training?
- The objective RAD's update optimizes falls consistently in training. For RAD-CVaR, the weighted FSD violation falls from 8.73, 8.55 and 8.28 at the start of training to 0.52, 0.59 and 0.35 at the end in the three seeds, with most of the decrease in the first 200 to 400 steps.
- The dual update then holds it at the constraint boundary. The update therefore consistently reduces the objective in practice, consistent with the ascent analysis in Appendix E.
Q2. Why the weighted entropic update?
- Unregularized OT gives a hard, nonsmooth particle update. RAD's hinge cost is not strictly convex, so Theorem 4 does not guarantee a unique optimal plan, and the particle subgradients change abruptly at quantile crossings (Appendix D).
- Entropic OT gives the update RAD uses. Its strictly convex penalty yields a unique soft transport plan, with particle gradients computed through Sinkhorn iterations (Appendix C).
Q3. What is the benefit of optimal transport over a simpler update?
- The reviewer does not specify which simpler update. The natural alternative without optimal transport is a direct CVaR constraint, which we trained with the Rockafellar–Uryasev objective and REINFORCE/RLOO in RAD's own pipeline (RLOO-CVaR), alongside Safe RLHF's expected-cost objective in the same setup (Safe-RLOO).
| Method | CVaR₀.₉ | CVaR₀.₉₅ | VaR₀.₉₅ | Safe (%) |
|---|---|---|---|---|
| Safe-RLOO | 1.60 ± 1.10 | 3.66 ± 1.38 | 1.04 ± 1.20 | 93.5 ± 1.6 |
| RLOO-CVaR | 0.71 ± 5.10 | 2.21 ± 5.14 | 0.14 ± 5.26 | 91.1 ± 7.5 |
| RAD-CVaR | −0.59 ± 0.30 | 1.11 ± 0.43 | −1.20 ± 0.17 | 97.0 ± 0.4 |
- RAD-CVaR has the best tail metrics and safe rate in the same pipeline. It also has far smaller seed variation than the direct CVaR constraint (CVaR₀.₉ ±0.30 against ±5.10), consistent with the regularizing effect of its entropic transport coupling.
- RAD also outperforms the CVaR-constrained Safe RLHF baseline at all three reported CVaR levels (Table 7).
On the score. The reviewer concedes that our CVaR baseline and tail-risk evaluation resolve several of their concerns. What remains are requests for further justification and unspecified additional experiments, and we have now run the comparison the review implies, against an update without optimal transport. None identifies an error in the method, and the reviewer rates the paper's soundness 2. Holding a clear rejection on these grounds contradicts the reviewer's own assessment, and would require rejecting any paper that leaves room for further justification. The reviewer acknowledges that several of their concerns were addressed, yet maintains the lowest possible rating. The rating is inconsistent with the review's own content, which undermines its reliability as an assessment of this paper. We ask the AC to determine, from the content of the review itself, whether the rating and the review retain any credibility.
Response by Authors [1/2]
W1. Theory of the policy update.
Answer. Each point holds RAD to a standard prior work does not meet.
Density factor. The omission is disclosed in Sec. 3, citing Li et al.'s (2024) density-free quantile estimator, as the review acknowledges. IQN (Dabney et al., ICML 2018) obtains CVaR-, Wang- and power-distorted policies by reweighting approximate quantiles without an ascent guarantee. The review nevertheless demands a guarantee absent from these methods. The revision supplies one for RAD's regularized FSD term (App. E). The density-free update ascends when per-quantile gradients are pairwise non-negatively aligned, regardless of their densities, or when inverse-density variation is below their gradient coherence. Equal densities guarantee ascent.
Entropic regularization. Like OT-based dominance methods (Farajzadeh et al., 2025), RAD uses transport for particle gradients (App. C). Entropy gives a unique, soft plan and well-defined particle gradients, converging to unregularized OT as χ→0 (Peyré & Cuturi, 2020, Prop. 4.1). At χ=0.01, cosine similarity with exact OT particle gradients across three seeds is at least 0.97 initially and above 0.9999 at the trained RAD-Uniform and RAD-CVaR checkpoints (App. E). In unregularized OT, the weighted derivative is exactly w(αi) times the unweighted derivative (App. E).
Dominance. Exact FSD requires an inequality at every quantile, so prior stochastic-dominance methods also use surrogates (Melnyk et al., 2024; Farajzadeh et al., 2025). RAD's surrogate attains empirical dominance. Across 7,495 held-out prompts, no reverse violation is detected for any variant or seed (App. H.1). The update therefore has stated ascent conditions, Corollary 3 bounds spectral-risk reduction, and the held-out data show empirical dominance. We ask that RAD be evaluated by the standard applied to established methods in this literature.
W2. Experimental validation.
Answer. Tail risk. Appendix H.1 adds full quantile curves and held-out CVaR₀.₈/₀.₉/₀.₉₅. RAD-CVaR is lowest at each level, −2.54/−0.59/1.09 versus Safe RLHF's −0.24/1.85/3.47. Its 95th-percentile cost is below the safety threshold, −1.20 versus 1.36. It retains a tail advantage under an independently trained cost model (App. H.2).
Risk-sensitive baseline. By Proposition 2, RAD is a spectral-risk-constrained Safe RLHF, with RAD-CVaR implementing the CVaR case and Wang and power corresponding to their weightings when reverse violation is zero. We additionally train Safe RLHF with the Rockafellar–Uryasev cost transform at the same 0.9 tail level and margin for three seeds. Both impose the same CVaR constraint at zero reverse violation (Corollary 3). RAD-CVaR has lower held-out CVaR at every reported level, −2.54/−0.59/1.09 versus −1.56/0.42/2.05 (App. H.1).
Optimizer ablation. This is RAD-Uniform, which uses the same pipeline without tail weighting. It matches Safe RLHF on every held-out cost statistic, while RAD-CVaR differs from it only in weighting and substantially lowers tail risk (App. H.1). The improvement comes from the objective rather than the optimizer.
Entropic regularization. Entropic regularization gives a unique, soft transport plan and well-defined particle gradients and recovers unregularized OT as χ→0. Training particle gradients align with exact OT gradients at cosine similarity at least 0.97 initially and above 0.9999 at the trained checkpoints (W1, App. E).
Density factors. The policy cost densities at each quantile are unknown and unavailable in closed form. RAD follows Li et al. (2024) in omitting them. Neither Li et al.'s estimator nor IQN is evaluated against a version with exact densities, because that version does not exist without first constructing a new density estimator. Appendix E instead proves when omission preserves ascent. The requested ablation demands a standard no method in this line meets.
Q1. Ascent and approximations. See W1. Appendix E proves two ascent conditions for the regularized FSD term. The entropic solution converges to unregularized OT as χ→0 (Peyré & Cuturi, 2020, Prop. 4.1), with measured particle-gradient cosine similarities of at least 0.97 initially and above 0.9999 at the trained checkpoints.
Q2. Weighted gradient. See W1. For unregularized OT, the weighted particle derivative is exactly w(αi) times the unweighted derivative. The entropic version uses the same smoothing approximation (App. E).
Q3. Tail comparisons, CVaR baseline and OT ablation. See W2. Appendix H.1 adds CVaR at three levels and the direct Rockafellar–Uryasev Safe RLHF baseline at the same tail level and margin. RAD-CVaR has lower cost at each level. Training gradients closely match exact OT particle gradients (W1, App. E).