- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes an RLHF method to address the safe alignment problems. Safety alignment of large language models aims to maximize the helpfulness under the safe constraints. Existing methods mainly focus on solving this problem via primal-dual methods, penalty-based methods, and primal methods. However, they have limitations because the safety risk is highly heterogeneous. This paper proposes prompt-level and response-level reweight methods to conduct policy optimization. Empirical results show improved performance compared to baselines.
Critical strengths
-
This paper aims to address a critical problem, which is RLHF to improve helpfulness and control safety. The proposed methods solve the existing methods' limitations and can be applied to heterogeneity across prompts and nonstationarity across training.
-
The empirical results illustrate the improved performance compared to baseline training with the PKU-SafeRLHF dataset evaluated on the in distribution set and out of disttribution benchmarks including XStest, HermBench, and StrongREJCT. Ablation study showing the improvements compared with the variants without the learning rate reweighting.
Critical weaknesses
-
It lacks a detailed explanation of the reasons to use the rollout-level calibration of learning rate. This design seems to be heuristic. The ablation study also shows that the improvement is marginal. The claim that there would be a large gradient with the new advantage should be supported by some empirical evidence.
-
RL for LLMs can be very noisy with limited responses. This means that for some prompts, they may not generate unsafe responses and will not be optimized with the safe constraints. Then it still has some risks of generating unsafe responses.
-
This paper lacks several important baselines to compare. For example, SafeDPO[1] and NSPO[2] are mentioned in this paper but not directly compared. It would be helpful to incorporate these two recent works.
[1] SafeDPO: A Simple Approach to Direct Preference Optimization with Enhanced Safety. ICLR 2026.
[2] Mitigating the Safety Alignment Tax with Null-Space Constrained Policy Optimization. ICLR 2026.
Questions for authors
-
Can you elaborate on the motivation for using the rollout-level calibration of learning rate?
-
Can you provide more empirical results to show that rollout-level calibration of learning rate is necessary to control the extremely large gradients?
-
Can you explain more about the design of the methods to avoid missing some prompts with unsafe problems?
-
Can you incoproate more baselines such as the SafeDPO[1] and NSPO[2]?
4 comments
Reply to Question 4: Additional baseline comparisons with SafeDPO and NSPO.
We thank the reviewer for suggesting these additional baselines. SafeDPO was not included in our original submission because its official implementation was not publicly available at the time, while NSPO introduces additional computational overhead through gradient projection. In response, we have now conducted additional experiments comparing HIPO with both SafeDPO and NSPO.
As shown in Table 6, HIPO achieves substantially higher helpfulness and harmlessness win rates against both baselines on PKU-SafeRLHF. On external benchmarks (Table 7), HIPO also achieves better XSTest refusal/over-refusal performance and lower HarmBench attack success rates.
Table R6. Pairwise GPT evaluation on PKU-SafeRLHF (Qwen3.5-9B).
| Comparison | Dimension | Mean scores | W/T/L (%) |
|---|---|---|---|
| HIPO vs. SafeDPO | Helpfulness | 6.862 vs. 5.810 | 77.00/2.33/20.67 |
| HIPO vs. SafeDPO | Harmlessness | 8.111 vs. 7.646 | 76.00/2.67/21.33 |
| HIPO vs. NSPO | Helpfulness | 6.845 vs. 5.438 | 80.00/2.00/18.00 |
| HIPO vs. NSPO | Harmlessness | 8.043 vs. 7.553 | 78.33/1.67/20.00 |
Table R7. GPT evaluation on external benchmarks (Qwen3.5-9B).
| Method | XSTest OR ↓ (%) | XSTest UR ↑ (%) | HarmBench ASR ↓ (%) | StrongREJECT score ↓ |
|---|---|---|---|---|
| SafeDPO | 4.00 | 68.53 | 37.92 | 0.0931 |
| NSPO | 6.80 | 70.00 | 36.67 | 0.0787 |
| HIPO | 1.08 | 84.87 | 35.29 | 0.1122 |
We acknowledge that SafeDPO and NSPO achieve lower StrongREJECT scores. This gap may partly reflect differences in response behavior. The StrongREJECT rubric assigns zero to refusals and otherwise evaluates how specifically and convincingly a response assists a harmful request. HIPO’s reward–cost objective may favor substantive responses with caution or redirection, but any remaining assistance toward the harmful objective still contributes to the score. Overall, these additional results further support HIPO’s effectiveness in balancing helpfulness and safety. We will incorporate both baselines into the revised manuscript.
We hope these clarifications and additional experimental results address the reviewer’s concerns and appriciate the reviewer to reevalute our paper. We would be happy to address any further questions.
Thank you for the detailed response. Part of my concerns have been resolved.
My main remaining concern is whether the calibration is needed in practice. Figure 5 shows how the learning rate changes, but it is still unclear to me how large the updates would be without calibration. My other concern is that four responses per prompt can miss rare unsafe responses. When this happens, the prompt receives no safety correction in that update.
Therefore I tend to maintain my current rating.
Thank for your response.
For your concern about the update magnitude without calibration, we did not record these results in our original experiments, as we mainly focused on the learning-rate ablation to see whether calibration helps achieve stable training and a better reward–cost trade-off. We would be happy to add the update magnitudes during training if you think this would be helpful. In fact, we have done a few preliminary checks and found that our method produces smaller updates. However, given the limited time, we would like to double-check these results before making a firm claim.
For your second concern, I think we should be optimistic on our model :-) As discussed in our rebuttal, if our model has already generated rate unsafe response, it implies that our model is safe already, where all safe signal can be also used to push our model to generate safe response, and we don't need to have the unsafe signal to avoid such unsafe behavior.
Hope these can address your concern. Thank you for your time.
We thank the reviewer for recognizing our contributions and providing constructive feedback. We believe that most of the concerns are clarification questions of our methodological design and additional baseline comparisons. We would like to eleborate as follows.
Reply to Questions 1 and 2: Motivation and effectiveness of rollout-level learning-rate calibration.
We thank the reviewer for the comments. We would like to clarify that the rollout-level learning-rate calibration is theoretically motivated, rather than simply a heuristic design. As illustrated in Eqs. (10)–(11) of the paper, we have
w_r(\eta)&=\frac{\alpha\sqrt{A}}{\sqrt{A+\eta^2}} \leq\alpha, &\qquad \lim_{\eta\to\infty}w_r(\eta)&=0,\\\\ w_c(\eta)&=\frac{\alpha\sqrt{A}\eta}{\sqrt{A+\eta^2}} \leq\alpha\sqrt{A}, &\qquad \lim_{\eta\to\infty}w_c(\eta)&=\alpha\sqrt{A}. \end{align}$$ Thus, the learning-rate-scaled safety coefficient remains bounded even as $\eta$ grows, preventing excessively large safety-weighted updates and promoting stable policy optimization to achieve a better reward–cost trade-off. More generally, Lemma 1 establishes that the proposed calibration uniformly bounds the scaled-gradient magnitude, independently of the current safety pressure, while preserving heterogeneous prompt-level safety weighting. This provides the main theoretical motivation for our design. Moreover, we did not observe gradient explosion or training instability in our experiments, consistent with the intended effect of the calibration. We respectfully disagree that the learning-rate calibration provides negligible improvements. As shown in Table 2, compared with a fixed learning rate, HIPO improves reward (5.3402 to 5.4240), reduces cost ($-6.2755$ to $-6.6993$), and improves safety rate ($94.33\\%$ to $96.00\\%$). Figure 5 further shows that HIPO reduces the learning rate under high safety pressure and gradually restores it as violations decrease. In hindsight, conventional warmup can be viewed as a predetermined approximation of this adaptive trajectory, improving reward and safety over the fixed learning rate. Nevertheless, HIPO achieves lower cost and higher safety than warmup, with a slight reward trade-off. **Reply to Question 3: Safety correction under limited response sampling.** We thank the reviewer for this comment. We would like to clarify that **not generating unsafe responses for certain prompts does not necessarily indicate a limitation of HIPO.** It may suggest that the model already has a good understanding of these prompts and can generate sufficiently safe responses. Moreover, even when no unsafe responses are sampled, **HIPO can still optimize these prompts through GRPO’s group-wise reward advantages**, encouraging the model to generate more helpful responses. Furthermore, HIPO employs group-wise sampling, where the probability of missing unsafe responses decreases as the group size increases. This is consistent with our group-size ablation (Table 8), where increasing $G$ from 2 to 8 improves the safety rate from $86.67\\%$ to $96.33\\%$. While finite sampling cannot completely eliminate the risk of missing rare unsafe behaviors, subsequent rollouts can identify and correct previously unobserved violations.