acceptodds
Under review as a conference paper at ICLR 2027

Towards Safety Alignment by Mitigating Imbalanced Preference Comprehension

Abstract

With the rapid development of Large Language Models (LLMs), their potential safety risks have attracted widespread attention. Reinforcement Learning from Human Feedback (RLHF) has been adopted to enhance the safety performance of LLMs. As a simple and effective alternative to RLHF, Direct Preference Optimization (DPO) is widely used for safety alignment. Despite the effectiveness of DPO for safety alignment, prior studies have identified overoptimization as a persistent issue in this method, which can compromise safety generalization. This paper explores this phenomenon from the perspective of the model’s comprehension of the training data. We find that the Imbalanced Preference Comprehension phenomenon exists between responses in preference pair, which compromises the model’s safety generalization. To address this issue, we propose Balanced Direct Preference Optimization (B-DPO), which adaptively modulates optimization strength between preferred and dispreferred responses based on an information-theoretic comprehension measure. Experiments across multiple LLMs demonstrate that B-DPO improves safety generalization over competitive preference optimization baselines while maintaining competitive general capabilities.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
Safety Alignment, Direct Preference Optimization, Overfitting, Balanced Preference Optimization

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_HZPfOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper observes the phenomenon of comprehension imbalance between the preferred and dispreferred responses within the same preference pair during DPO-based safety alignment. Based on this observation, the paper further proposes B-DPO, which adjusts the optimization strength assigned to the two responses to mitigate overfitting and improve safety generalization. Experiments on eight safety and general capability benchmarks show that the proposed B-DPO consistently achieves the best overall safety performance while maintaining competitive general capabilities.

Critical strengths

1. Insightful observation on imbalanced preference comprehension. This paper reveals an interesting phenomenon in DPO-based safety alignment: within the same preference pair, the model may exhibit substantially different levels of comprehension for the preferred and dispreferred responses.

2. The quantitative formulation of comprehension imbalance. This paper introduces an information-theoretic measure to quantify the model’s comprehension of individual responses, providing a concrete way to characterize the degree of comprehension imbalance within each preference pair.

3. Simple and effective method design. The proposed B-DPO adjusts the optimization strength assigned to the two responses according to their comprehension levels and further introduces gradient correction to preserve the overall gradient coefficient of standard DPO.

Critical weaknesses

1. Incremental novelty and unclear necessity of response-specific weighting. The core design of B-DPO is to perform response-specific adaptive weighting within each preference pair, using β⋅λw\beta \cdot \lambda_{w} and β⋅λl\beta \cdot \lambda_{l} to separately control the optimization strengths of the preferred and dispreferred responses. However, it remains unclear whether such asymmetric response-level weighting is necessary. In particular, it is unclear whether the proposed comprehension signal could instead be used to determine a single adaptive β\beta for each preference pair and achieve comparable performance without introducing response-specific weighting.

2. Insufficient evaluation of model general capabilities. As discussed in Section 5.2, the authors claim that B-DPO maintains competitive general capabilities. To better support this claim, the evaluation should not only use the unsafe split of XsTest to assess safety, but also include the safe split to test whether B-DPO introduces additional over-refusal issues compared with the base model, i.e., whether the model unnecessarily refuses benign prompts. Such evaluations would provide a more complete assessment of the trade-off between safety improvement and model usability.

3. Limited diversity of safety evaluators. Most safety evaluations rely on Qwen3-14B-Instruct as the LLM-as-a-judge, except for HarmBench, which uses its official classifier. However, the paper provides limited justification for choosing Qwen3-14B-Instruct as the primary safety evaluator. Recent guard-specific models may provide more targeted safety judgments. You might want to include other guard-specific models as independent judges and report whether the results remain consistent across different evaluators for further validation.

Questions for authors
  1. The authors should compare B-DPO with the pair-level adaptive β\beta method, i.e., β\beta -DPO [1]. Such a comparison would help determine whether the proposed response-specific weighting provides additional benefits that cannot be achieved through pair-level adaptation alone, and would better establish the necessity and contribution of B-DPO's asymmetric weighting design.

  2. The authors should provide additional evaluations on over-refusal benchmarks, such as the safe split of XsTest and ORBench [2], comparing B-DPO with the base model. This would help verify whether B-DPO improves safety without introducing additional refusals on benign prompts, thereby providing stronger evidence that it preserves model general capabilities and usability.

  3. The authors should include additional guard-specific models, such as WildGuard [3] and Qwen3Guard [4], as independent safety evaluators and report whether the performance trends of B-DPO remain consistent across different judges. This would provide stronger evidence for the robustness and reliability of the reported safety improvements.

[1] Junkang Wu et al., β\beta-DPO: Direct Preference Optimization with Dynamic β\beta, NeurIPS 2024.

[2] Justin Cui et al., OR-Bench: An Over-Refusal Benchmark for Large Language Models, ICML 2025.

[3] Seungju Han et al., WildGuard: Open One-Stop Moderation Tools for Safety Risks, Jailbreaks, and Refusals of LLMs, NeurIPS 2024.

[4] Haiquan Zhao et al., Qwen3Guard Technical Report, arXiv 2025.

Reviewer_6WEmOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies the imbalance between preferred and rejected responses in DPO-based safety alignment. The authors use an entropy-based measure to estimate how well a model understands each response. Based on this measure, they propose B-DPO, which adjusts the optimization weights of the two responses and adds a gradient correction term. Experiments on three LLMs show better safety performance than the tested baselines, with limited loss in general capabilities.

Critical strengths
  1. Interesting observation. The paper studies how models treat preferred and rejected responses differently. The results in Figure 2 show clear differences across harmful and harmless prompts. This is a useful observation for safety alignment.

  2. Good empirical results. B-DPO improves safety scores across three models and four safety benchmarks. The authors also provide ablation studies and results from multiple runs, which help support the reported gains.

Critical weaknesses
  1. Figure 2 shows an imbalance, but it is unclear why this imbalance is harmful. Figure 3 only shows a correlation with worse performance. Other factors, such as response length or data difficulty, may explain the results. There is no clear evidence that the imbalance itself causes overfitting.

  2. Balancing the optimization of chosen and rejected responses has been studied before. For example, Ma et al. (2025) proposed Balanced-DPO, and Cho et al. (2025) proposed Bounded-DPO. The proposed weighting rule is different, but the main idea is similar. Comparisons with these methods are missing.

  3. The entropy difference in Eq. (5) does not directly measure model comprehension. Also, the correction in Eq. (10) only restores a scalar gradient factor, not the full gradient magnitude. It is unclear why these choices should reduce overfitting.

Questions for authors
  1. Is the observed imbalance really harmful? Could it simply come from natural differences between safe and unsafe responses? Can the authors show that reducing this imbalance improves generalization when response length and data difficulty are controlled? A simple weighting baseline based on harmful/harmless prompt types would also help.

  2. How does B-DPO compare with Balanced-DPO (Ma et al., 2025) and Bounded-DPO (Cho et al., 2025)?

  3. Can the authors compare the proposed comprehension measure with simpler measures, such as token entropy or response likelihood? Why is this measure more suitable?

  4. In Table 2, DPO+GC should be equivalent to DPO when the optimization weights are removed. Why are their results different, especially on AdvBench (80.61% vs. 73.80%)?

Reviewer_VwUVOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper investigates the imbalance in model comprehension between preferred and dispreferred responses in safety preference datasets. The authors propose Balanced Direct Preference Optimization (B-DPO), which uses a mutual-information-based measure to adjust the optimization weights of the two responses. Experiments on three LLMs demonstrate improvements over several DPO baselines on safety benchmarks.

Critical strengths
  • The paper investigates an interesting phenomenon in preference optimization and provides empirical observations regarding the imbalance between preferred and dispreferred responses. B-DPO is relatively simple to implement and achieves improvements on several safety benchmarks.
Critical weaknesses
  1. The paper's motivation is that imbalanced preference comprehension leads to poor safety generalization. However, the proposed mutual information essentially measures the reduction in predictive uncertainty when conditioning on a prompt. It is not clear why this quantity appropriately captures model comprehension. It would require more explanation.

  2. The proposed B-DPO method essentially reweights the gradients associated with preferred and dispreferred responses. Similar ideas of adjusting optimization strength based on model confidence, preference difficulty, or response characteristics have been extensively explored in preference optimization. The ablation studies only examine the proposed weighting mechanism and gradient correction factor. Besides, it is unclear why the authors use the proposed mutual information rather than simpler measures, such as conditional entropy H(y∣x)H(y | x) or response likelihood. Comparisons with these alternatives are necessary to establish the effectiveness of the proposed design

  3. Although the paper is motivated by safety alignment, the proposed method appears to be a general preference optimization technique. Unlike safety alignment approaches that explicitly distinguish helpfulness and harmlessness objectives, B-DPO does not model their potential conflict or exploit safety-specific optimization structure. It is therefore unclear what makes the proposed method particularly suitable for safety alignment rather than general preference optimization.

  4. The paper primarily evaluates safety alignment on harmful prompts, without examining whether the proposed method increases over-refusal on benign requests. In particular, the experiments use only the unsafe split of XsTest, although this benchmark is designed to assess exaggerated safety behaviors. Improvements in refusal-based safety metrics do not necessarily imply better safety alignment, as they may come at the cost of rejecting legitimate requests. The authors should evaluate over-refusal and the trade-off between helpfulness and harmlessness to better support their claims regarding safety generalization.

Questions for authors

Please see weakness above.

3 comments

We sincerely thank you for your valuable suggestions. Over the past few days, we have conducted extensive additional experiments to address your concerns. Below are our detailed responses.

for question1:

We appreciate your insightful interpretation and agree that this metric directly reflects the average reduction in response uncertainty given the prompt. A higher value indicates a stronger association between the prompt and response, which we intuitively refer to as comprehension To provide further evidence, we analyzed the gradients of the response log-probability with respect to the model parameters and observed a clear negative correlation with mutual information, as shown in the Figure 1 at the anonymous link https://anonymous.4open.science/r/iclr_rebuttal_figures-347C/

This suggests that samples with higher mutual information lie in flatter regions of the optimization landscape, indicating that the model's learning of these samples is relatively saturated. We consider this a potential explanation for why this metric captures the model's comprehension.

for question2:

We conducted additional experiments by replacing mutual information with H(y∣x) and the average logp(y∣x). For the former, responses with higher H(y∣x) are assigned larger weights. For the latter, the side with a higher average ∣logp(y∣x)∣ is assigned a larger weight.

Qwen:

MethodGSM8kXsTest unsafeStrongRejectHarmBenchHHHAdvGLUETruthfulQAXsTest safe
H(y∣x)72.98%84.00%86.54%80.67%82.35%71.41%48.96%6.80%
ave ∣logp(y∣x)∣73.54%81.50%87.22%81.33%82.35%71.95%48.84%5.60%
Ours73.54%83.08%88.91%82.78%83.18%71.25%49.29%4.80%

Mistral:

MethodGSM8kXsTest unsafeStrongRejectHarmBenchHHHAdvGLUETruthfulQAXsTest safe
H(y∣x)54.13%80.00%88.78%51.33%84.62%60.84%39.78%4.40%
ave ∣logp(y∣x)∣56.71%80.50%90.68%54.33%84.62%62.47%39.78%4.80%
Ours53.54%83.33%92.32%59.11%84.46%62.44%37.82%4.00%

Vicuna:

MethodGSM8kXsTest unsafeStrongRejectHarmBenchHHHAdvGLUETruthfulQAXsTest safe
H(y∣x)12.74%71.00%67.41%52.33%87.26%43.90%34.76%7.60%
ave ∣logp(y∣x)∣11.45%70.00%63.55%52.00%87.44%43.09%34.52%7.60%
Ours13.82%76.08%73.61%56.56%93.21%43.27%34.43%6.40%

The results show that neither metric yields a significant improvement in safety, while both are more prone to over-refusal than our chosen metric.Following our explanation for Weakness 1, we further analyzed their relationships with the gradients of the average response log-probability and posterior entropy. The results is also in the anonymous link and there is no monotonicity between them(Figure2 and Figure3). This suggests that our metric better identifies samples that have not been sufficiently learned by the model. As stated in the paper, selecting mutual information as our metric is also intended to exclude the influence of the response itself on generation uncertainty. Furthermore, in the anonymous link above, we provide a correlation analysis between 𝐻(𝑦) and 𝐻(𝑦∣𝑥)(Figure 4). The results reveal a strong correlation between the two, which is one of the key reasons we ultimately chose mutual information.

for question4:

We evaluated our method and other baselines on the safe subset of XSTest. We report the refusal rate,The results are shown below:

XsTest-safe:

BaseDPOcDPOFocalPOSafeDPOOurs
Qwen4.40%5.33%4.60%4.80%4.80%4.80%
Mistral2.40%3.87%4.53%3.47%3.87%4.00%
Vicuna3.60%5.87%5.60%8.40%6.93%6.40%

While our method does not yield the lowest over-refusal rate, it offers a clear safety advantage without incurring substantially more over-refusal than baselines, thus maintaining a solid trade-off. To explicitly address your concern on benign requests, experiments on the Hard-1K subset of OR-Bench (designed to test exaggerated safety) are still in progress. We will upload the results once completed and sincerely appreciate your understanding of our limited time.

for question3:

We acknowledge that B-DPO is highly general in its method design. However, as shown in Figure 2, we observed a distinct distribution difference between harmful and harmless data under our selected metric. This implies a unique optimization conflict in safety alignment. Specifically, Figure 2 indicates that on the harmful dataset, the mutual information on the safe side is excessively large. Our method dynamically balances this conflict by suppressing this specific signal during optimization, which likely explains why our approach avoids excessive over-refusal while maintaining safety. Therefore, we believe it is well-justified to apply this method to safety alignment.

for question4:

We have added the results on the Hard-1K subset of OR-Bench, as shown in the table below:

BaseDPOcDPOFocalPOSafeDPOOurs
Qwen29.87%46.43%39.8%44.46%53.16%50.87%
Mistral3.06%7.51%10.45%10.03%14.00%8.55%
Vicuna12.91%29.06%26.07%35.04%33.48%32.75%

Similar to the results on the safe subset of XSTest, our method does not exhibit the highest level of over-refusal among the baselines. However, we acknowledge that it shows some degree of over-refusal compared with standard DPO. Considering both safety and general performance, we believe that our method achieves a reasonable trade-off between safety and general capabilities.

Finally, if you have any further questions, we will do our best to address your concerns.

Dear Reviewer,

We would like to follow up on our response submitted a week ago. We have provided additional experiments and further clarifications to address your questions and concerns. The experimental results are consistent with and further support our conclusions.

We hope you will have a chance to review our response and provide your feedback on the questions we raised. Your response would help us better understand whether our clarifications and additional experiments have adequately addressed your concerns.

Thank you for your time.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.