acceptodds
Under review as a conference paper at ICLR 2027

The Real Conflict of Multi-objective Alignment Is Hidden in the Preference Labels

Abstract

Multi-objective alignment seeks to reconcile conflicts among helpfulness, safety, and honesty. However, our experiments show that these conflicts can depend on the training signal: they emerge in DPO training on preference data, whereas training guided by explicit reward functions can improve these qualities simultaneously. This discrepancy points to a hidden preference conflict: learning harmful compliance or sycophancy encouraged by the labels can undermine safety or honesty. These behavioral incentives are implicit in the preference data rather than specified as separate rewards. We propose Conflict-Disentangled Direct Preference Optimization (CD-DPO), which learns a weighted combination of frozen reward-model scores from examples annotated for these behaviors and uses it to correct preference labels before standard DPO training. Allowing negative weights lets the method identify unwanted behaviors through differences between reward scores rather than simply adding them. In a stylized model, we prove that allowing negative weights can suppress these behaviors more effectively than nonnegative reward scalarization at the same KL budget, and quantify the effects of fitting and training errors. Compared with vanilla DPO, CD-DPO achieves large safety gains on Qwen2.5 (0.5B, 3B, and 7B) and improves honesty and truthfulness on both Qwen2.5 and Mistral-7B at the same policy-training budget.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
multi-objective alignment

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_zPxVOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper argues that preference labels can push models toward bad behaviors, like following harmful requests, agreeing with wrong user beliefs, or making things up. This can happen even when the reward scores look positively correlated. To fix it, the authors train a simple linear classifier on reward model scores to spot the bad behavior. The weights can be negative. They flip the pairs where the chosen answer looks worse, and then run normal DPO. They also give a theory for why negative weights help. On Qwen2.5 and Mistral-7B with made-up contaminated data, the method improves safety, honesty, and truthfulness.

Critical strengths
  1. Clean diagnosis of where the conflict lives

Rewards like helpfulness and safety are positively correlated, yet DPO on certain labels still hurts safety. The +paired vs. +hidden experiments show this for all three behaviors, at three model sizes, and with independent evaluators. I think this is a useful point for the multi-objective alignment community. Using negative weights to subtract out shared quality is also a simple and sensible idea.

  1. Simple method, with theory and careful reporting

The method only adds a dot product per pair before standard DPO, so it is easy to use. The theory explains why negative weights help under the same KL. The authors also report the cases where the method does not clearly win, such as blind reversal in the pure setting and the honesty results across seeds, which makes me trust the results more.

Critical weaknesses
  1. Missing baselines that actually do label correction

The paper argues that flipping labels beats loss-side fixes, but the baselines (fixed margins, cDPO, IPO, SimPO, BFPO) don’t do per-pair label correction. Appendix G’s argument only covers additive margins. Methods like rDPO (Chowdhury et al., ICML 2024) and RE-PO (Cao et al., ICLR 2026) estimate per-pair label reliability and can effectively reverse the gradient when confidence is low. In the 33% mixture setting, corrupted pairs are the minority and conflict with the clean majority, so consistency-based methods have a real shot at catching them without any external annotation. So I'm not sure whether CD-DPO’s gains come from the act of flipping itself or from the extra behavioral annotations. I’d like to see such baselines added.

  1. The conflict is manufactured, and the method needs to know what to look for

All experiments inject contamination by construction. Do real preference datasets (e.g., UltraFeedback or HH-RLHF) actually contain this kind of hidden conflict? The paper’s own calibration in Appendix A.1 doesn’t really suggest so. On top of that, CD-DPO requires labeled examples of the specific bad behavior to fit the predictor, so the behavior must be known and annotatable upfront. This sits awkwardly with the “hidden” framing in the title because in practice we often don’t know which bad behaviors are lurking in the data.

Questions for authors
  1. How does CD-DPO compare to label-correcting robust DPO methods such as rDPO or RE-PO?

  2. When the predictor is applied to an unmodified real dataset (e.g., UltraFeedback), what do the flipped pairs actually look like?

Reviewer_BSGAOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

The paper argues that reward scores and preference labels measure different things: ArmoRM helpfulness correlates positively with safety, honesty and truthfulness, yet DPO on labels that favor harmful compliance, sycophancy or hallucination lowers the paired reward. The authors call this a hidden preference conflict and propose CD-DPO, which fits a sign-unrestricted logistic-regression predictor of the unwanted behavior on the 19 frozen ArmoRM scores, reverses every pair whose chosen answer scores higher on that predictor, and then runs ordinary DPO. A stylized log-linear policy model shows that a predictor ordering pairs better than any same-sign predictor suppresses the behavior more at equal steps and KL. On Qwen2.5-3B, safety rises from 0.033 to 0.999 under pure contamination and from 0.139 to 0.897 under 33% contamination.

Critical strengths
  • the correlated-rewards versus conflicting-labels dissociation, supported by the three behaviors and three scales.
  • Negative weights matter empirically: task AUC is 0.892 vs 0.645, 0.907 vs 0.500 (sycophancy) and 0.860 vs 0.542 (hallucination) with and without the nonnegativity constraint.
  • Large effect sizes on the headline metric: safety 0.033 to 0.999 on 230 prompts (Table 1 DPO and CD-DPO rows) and 0.139 to 0.897 under 33% contamination, with CD-DPO also beating pair removal, MODPO-style margins, BFPO, IPO, SimPO, conservative DPO and judge relabeling in the main mixed runs (Table 2).
Critical weaknesses
  1. The hidden conflict is created by the authors rather than found in real preference data, so the title claim ("the real conflict ... is hidden in the preference labels") is not supported. The harmful-compliance, sycophancy and hallucination datasets are constructed so that every chosen answer exhibits the unwanted behavior, and the mixed setting injects 33% of them into UltraFeedback. In the pure case flipping every pair is as good as CD-DPO (blind 1.002 vs 0.999), so the pure result says nothing about the predictor. On clean UltraFeedback the predictor reverses 32.6% of pairs, and the dataset-level check threshold of 0.35 was calibrated on the same few measured datasets, with 0.336 (clean HH-RLHF) and 0.394 (10% contamination) on either side of it. No experiment shows that a naturally collected preference set contains such a conflict.

  2. Evaluation is partly circular and mostly relies on small samples. The predictor is built from ArmoRM scores, and the headline numbers are ArmoRM rewards, so the method is optimized against the metric. Independent evidence is thinner: Beaver-Dam is reported for one comparison, 0.12 to 0.51 (Table 3), the DeepSeek-V3 judge covers only honesty and truthfulness, and a local 7B judge inverts the honest-mixture ranking. Evaluation sets are small: 230 safety prompts, 51 honesty pairs and 200 truthfulness pairs, and the main runs use a single seed (seed 0). The Mistral-7B replication shows helpfulness-oriented DPO lowering safety (0.453 to 0.320), which also weakens the claim that quality-oriented DPO leaves the paired reward unchanged. The 7B LoRA helpful-DPO raises safety from 0.451 to 0.707, also at odds with that statement.

  3. The honesty improvement over reversing every pair is not reliable, yet it appears as a contribution. The text itself says the advantage varies with seed and fit split. Five further seeds give 0.554-0.574 against blind 0.578, a fit on all 150 pairs gives 0.446 (below vanilla 0.454), and the bootstrap intervals overlap. The honesty predictor is fitted on 112 pairs and validated on 38 . The truthful-axis edge over blind has a gap of 0.001 between intervals, unadjusted for multiplicity, and the sequential baseline beats CD-DPO on truthfulness by 0.043.

Questions for authors

N/A

1 comment

Dear Reviewer BSGA,

As the discussion period is nearing its end, we would like to send a gentle reminder regarding our response and the revised manuscript. If your remaining question has been resolved, we would greatly appreciate if you could consider raising your score. Please let us know if anything remains unclear.

Thank you again for your time and thoughtful feedback.

Reviewer_fscHOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper distinguishes correlations among reward-model scores from the behavioral incentives encoded by preference labels. It argues that preferences favoring harmful compliance, sycophancy, or hallucination can undermine alignment even when reward scores are positively correlated. Conflict-Disentangled DPO (CD-DPO) fits a signed linear combination of frozen reward scores using behavior annotations, selectively reverses preference labels, and then applies standard DPO. A stylized analysis relates improved pair ordering to suppression of unwanted behavior under stated assumptions. Experiments examine constructed and mixed preference contamination across several models, with the strongest results in safety and more qualified results in honesty and truthfulness. The evidence supports selective correction of contaminated preferences more directly than a general explanation of multi-objective conflict.

Critical strengths
  1. A useful distinction with an implementable correction method. A response’s reward scores and the choice of which response is preferred are different objects; the paper makes this distinction explicit and uses behavior annotations to turn it into a practical relabeling procedure. Algorithm 1 and the predictor definition make the intervention reproducible at the conceptual level. This provides a concrete tool for preference-data auditing rather than only a qualitative critique of alignment objectives.

  2. Substantial safety gains under mixed contamination, supported by meaningful controls. In the 33% mixed setting of Table 2, in-distribution safety is 0.897 for CD-DPO, compared with 0.396 for blind reversal and 0.620 for judge relabeling. The paper also includes independent evaluation, single-head and direction controls, and additional seeds. These controls should not be overlooked: they provide a real acceptance case for a focused paper on correcting contaminated preference data.

Critical weaknesses
  1. The central explanatory claim is not established by the current experimental design. The constructed datasets deliberately prefer harmful or otherwise undesirable responses, so degradation under uncorrected DPO and improvement after correction do not establish that label contamination is the general source of multi-objective conflict in naturally occurring preferences. In addition, the theoretical comparison covers all same-sign predictors, including all-nonpositive directions, whereas the main empirical contrast emphasizes unrestricted versus nonnegative predictors. Thus, the experiments do not fully establish the necessity of mixed-sign combinations relative to the theory’s comparison class. This matters because the paper’s distinctive explanatory claim goes beyond showing that behavior-supervised label correction can work.

  2. The evidence for a consistent advantage across behaviors is limited by instability and overlapping questions. The honesty headline reports 0.603 versus 0.578 for blind reversal, but five additional CD-DPO seeds yield 0.554–0.574, below the reported blind point estimate; this does not establish a matched multi-seed defeat, but it does weaken the headline advantage. TruthfulQA also has question-level overlap between predictor fitting and generation evaluation, as acknowledged in the appendix; pair-level disjointness does not remove that dependence. These limitations make it difficult to interpret the safety, honesty, and truthfulness results as comparably robust evidence for one general mechanism.

Questions for authors
  1. Clarification: Is the intended claim that CD-DPO repairs behavior-contaminated preferences, or that such contamination explains the “real conflict” in naturally occurring multi-objective alignment? What evidence in the current paper distinguishes the latter from genuine tradeoffs among desired objectives?

  2. Additional evidence that could change my rating: Under the same fitting split, behavior-label budget, and policy-training conditions, how do unrestricted, all-nonnegative, all-nonpositive, and single-head predictors compare on held-out pair-ordering accuracy and downstream behavior? Please also clarify how the p.30 single-head out-of-distribution results map to the main-table checkpoints and conditions.

  3. Additional evidence that could change my rating: Does the honesty advantage persist under a fixed fitting protocol and paired training seeds for CD-DPO and the strongest simple relabeling baseline? For truthfulness, are question-disjoint results available that isolate predictor fitting and training from final evaluation? These checks directly affect the claimed breadth of the method, rather than serving as general future-work suggestions.

1 comment

Dear Reviewer fscH,

As the discussion period is nearing its end, we would like to send a gentle reminder regarding our response and the revised manuscript. If your remaining question has been resolved, we would greatly appreciate if you could consider raising your score. Please let us know if anything remains unclear.

Thank you again for your time and thoughtful feedback.

Comments on the paper

1 comment

Dear PCs, SACs, ACs, and reviewers for their time and efforts.

We sincerely thank you and the reviewers for your time and constructive feedback. We would like to briefly highlight several points that we believe are important for the final assessment of our submission.

The Problem We Identify. Multi-objective alignment aims to reconcile helpfulness, safety, and honesty. A common assumption is that if the rewards for these objectives are positively correlated, training on preference data should improve them jointly. We show this assumption is wrong. DPO training on different preference categories can have markedly different effects even when their rewards are positively correlated. The reason: preference labels themselves implicitly incentivize unwanted behaviors—harmful compliance, sycophancy, and hallucination—that are never specified as separate rewards. We call this a hidden preference conflict.

Crucially, this is not an artifact of constructed data. On clean UltraFeedback and HH-RLHF, roughly one third of preference pairs favor an unwanted behavior. Correcting them improves safety at negligible helpfulness cost.

Our Method: CD-DPO. We propose Conflict-Disentangled Direct Preference Optimization (CD-DPO). The key insight is that unwanted behaviors can be identified through differences between reward-model scores, not by simply adding them up. Concretely:

  1. We fit a linear predictor on frozen reward-model scores using a small annotated set, allowing negative weights so the predictor can exploit reward differences to isolate behavioral information from shared quality information.
  2. We use this predictor to correct preference labels—reversing pairs where the chosen response shows stronger evidence of the hidden behavior—before running standard DPO.
  3. The correction costs only O(m) arithmetic operations per pair with cached reward vectors, and retains the efficiency of offline DPO training.

Our Theoretical Contribution. In a stylized model, we prove that a fitted predictor whose pair ordering is more accurate than every same-sign predictor yields greater suppression of the unwanted behavior at the same number of DPO updates and KL divergence. We also characterize the fitting requirements and the effects of training errors. This formally explains why negative weights matter.

Our Empirical Results · Large safety gains on Qwen2.5 (0.5B, 3B, and 7B) compared with DPO on original labels. · Improved honesty and truthfulness on both Qwen2.5 and Mistral-7B at the same policy-training budget. · The mechanism is architecture-independent: it replicates on the raw Mistral-7B base. · End-to-end correction on natural datasets (UltraFeedback, HH-RLHF) raises safety by +0.114 and +0.141 at essentially zero helpfulness cost, with an independent judge rating the corrected (rejected) responses safer or more honest in 78% of flipped pairs. And we have provided additional ablations, expanded benchmark evaluations, and statistical analyses.

We reveal that the real conflict in multi-objective alignment is hidden in the preference labels themselves—labels implicitly reward harmful compliance, sycophancy, and hallucination—and we propose CD-DPO, which uses negative-weight reward differences to identify and correct these labels, delivering large safety, honesty, and truthfulness gains at negligible helpfulness cost.

We respectfully hope that the PCs, SACs, and ACs will consider both the original submission and the substantial additional evidence provided during the rebuttal when making the final decision. We believe the work offers a valuable contribution to the LLM alignment area.

Thank you again for your consideration.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.