- Rating
- 2
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
This paper observes the phenomenon of comprehension imbalance between the preferred and dispreferred responses within the same preference pair during DPO-based safety alignment. Based on this observation, the paper further proposes B-DPO, which adjusts the optimization strength assigned to the two responses to mitigate overfitting and improve safety generalization. Experiments on eight safety and general capability benchmarks show that the proposed B-DPO consistently achieves the best overall safety performance while maintaining competitive general capabilities.
Critical strengths
1. Insightful observation on imbalanced preference comprehension. This paper reveals an interesting phenomenon in DPO-based safety alignment: within the same preference pair, the model may exhibit substantially different levels of comprehension for the preferred and dispreferred responses.
2. The quantitative formulation of comprehension imbalance. This paper introduces an information-theoretic measure to quantify the model’s comprehension of individual responses, providing a concrete way to characterize the degree of comprehension imbalance within each preference pair.
3. Simple and effective method design. The proposed B-DPO adjusts the optimization strength assigned to the two responses according to their comprehension levels and further introduces gradient correction to preserve the overall gradient coefficient of standard DPO.
Critical weaknesses
1. Incremental novelty and unclear necessity of response-specific weighting. The core design of B-DPO is to perform response-specific adaptive weighting within each preference pair, using and to separately control the optimization strengths of the preferred and dispreferred responses. However, it remains unclear whether such asymmetric response-level weighting is necessary. In particular, it is unclear whether the proposed comprehension signal could instead be used to determine a single adaptive for each preference pair and achieve comparable performance without introducing response-specific weighting.
2. Insufficient evaluation of model general capabilities. As discussed in Section 5.2, the authors claim that B-DPO maintains competitive general capabilities. To better support this claim, the evaluation should not only use the unsafe split of XsTest to assess safety, but also include the safe split to test whether B-DPO introduces additional over-refusal issues compared with the base model, i.e., whether the model unnecessarily refuses benign prompts. Such evaluations would provide a more complete assessment of the trade-off between safety improvement and model usability.
3. Limited diversity of safety evaluators. Most safety evaluations rely on Qwen3-14B-Instruct as the LLM-as-a-judge, except for HarmBench, which uses its official classifier. However, the paper provides limited justification for choosing Qwen3-14B-Instruct as the primary safety evaluator. Recent guard-specific models may provide more targeted safety judgments. You might want to include other guard-specific models as independent judges and report whether the results remain consistent across different evaluators for further validation.
Questions for authors
-
The authors should compare B-DPO with the pair-level adaptive method, i.e., -DPO [1]. Such a comparison would help determine whether the proposed response-specific weighting provides additional benefits that cannot be achieved through pair-level adaptation alone, and would better establish the necessity and contribution of B-DPO's asymmetric weighting design.
-
The authors should provide additional evaluations on over-refusal benchmarks, such as the safe split of XsTest and ORBench [2], comparing B-DPO with the base model. This would help verify whether B-DPO improves safety without introducing additional refusals on benign prompts, thereby providing stronger evidence that it preserves model general capabilities and usability.
-
The authors should include additional guard-specific models, such as WildGuard [3] and Qwen3Guard [4], as independent safety evaluators and report whether the performance trends of B-DPO remain consistent across different judges. This would provide stronger evidence for the robustness and reliability of the reported safety improvements.
[1] Junkang Wu et al., -DPO: Direct Preference Optimization with Dynamic , NeurIPS 2024.
[2] Justin Cui et al., OR-Bench: An Over-Refusal Benchmark for Large Language Models, ICML 2025.
[3] Seungju Han et al., WildGuard: Open One-Stop Moderation Tools for Safety Risks, Jailbreaks, and Refusals of LLMs, NeurIPS 2024.
[4] Haiquan Zhao et al., Qwen3Guard Technical Report, arXiv 2025.
We sincerely thank you for your valuable suggestions. Over the past few days, we have conducted extensive additional experiments to address your concerns. Below are our detailed responses.
for question1:
We appreciate your insightful interpretation and agree that this metric directly reflects the average reduction in response uncertainty given the prompt. A higher value indicates a stronger association between the prompt and response, which we intuitively refer to as comprehension To provide further evidence, we analyzed the gradients of the response log-probability with respect to the model parameters and observed a clear negative correlation with mutual information, as shown in the Figure 1 at the anonymous link https://anonymous.4open.science/r/iclr_rebuttal_figures-347C/
This suggests that samples with higher mutual information lie in flatter regions of the optimization landscape, indicating that the model's learning of these samples is relatively saturated. We consider this a potential explanation for why this metric captures the model's comprehension.
for question2:
We conducted additional experiments by replacing mutual information with H(y∣x) and the average logp(y∣x). For the former, responses with higher H(y∣x) are assigned larger weights. For the latter, the side with a higher average ∣logp(y∣x)∣ is assigned a larger weight.
Qwen:
Mistral:
Vicuna:
The results show that neither metric yields a significant improvement in safety, while both are more prone to over-refusal than our chosen metric.Following our explanation for Weakness 1, we further analyzed their relationships with the gradients of the average response log-probability and posterior entropy. The results is also in the anonymous link and there is no monotonicity between them(Figure2 and Figure3). This suggests that our metric better identifies samples that have not been sufficiently learned by the model. As stated in the paper, selecting mutual information as our metric is also intended to exclude the influence of the response itself on generation uncertainty. Furthermore, in the anonymous link above, we provide a correlation analysis between 𝐻(𝑦) and 𝐻(𝑦∣𝑥)(Figure 4). The results reveal a strong correlation between the two, which is one of the key reasons we ultimately chose mutual information.
for question4:
We evaluated our method and other baselines on the safe subset of XSTest. We report the refusal rate,The results are shown below:
XsTest-safe:
While our method does not yield the lowest over-refusal rate, it offers a clear safety advantage without incurring substantially more over-refusal than baselines, thus maintaining a solid trade-off. To explicitly address your concern on benign requests, experiments on the Hard-1K subset of OR-Bench (designed to test exaggerated safety) are still in progress. We will upload the results once completed and sincerely appreciate your understanding of our limited time.