- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper argues that preference labels can push models toward bad behaviors, like following harmful requests, agreeing with wrong user beliefs, or making things up. This can happen even when the reward scores look positively correlated. To fix it, the authors train a simple linear classifier on reward model scores to spot the bad behavior. The weights can be negative. They flip the pairs where the chosen answer looks worse, and then run normal DPO. They also give a theory for why negative weights help. On Qwen2.5 and Mistral-7B with made-up contaminated data, the method improves safety, honesty, and truthfulness.
Critical strengths
- Clean diagnosis of where the conflict lives
Rewards like helpfulness and safety are positively correlated, yet DPO on certain labels still hurts safety. The +paired vs. +hidden experiments show this for all three behaviors, at three model sizes, and with independent evaluators. I think this is a useful point for the multi-objective alignment community. Using negative weights to subtract out shared quality is also a simple and sensible idea.
- Simple method, with theory and careful reporting
The method only adds a dot product per pair before standard DPO, so it is easy to use. The theory explains why negative weights help under the same KL. The authors also report the cases where the method does not clearly win, such as blind reversal in the pure setting and the honesty results across seeds, which makes me trust the results more.
Critical weaknesses
- Missing baselines that actually do label correction
The paper argues that flipping labels beats loss-side fixes, but the baselines (fixed margins, cDPO, IPO, SimPO, BFPO) don’t do per-pair label correction. Appendix G’s argument only covers additive margins. Methods like rDPO (Chowdhury et al., ICML 2024) and RE-PO (Cao et al., ICLR 2026) estimate per-pair label reliability and can effectively reverse the gradient when confidence is low. In the 33% mixture setting, corrupted pairs are the minority and conflict with the clean majority, so consistency-based methods have a real shot at catching them without any external annotation. So I'm not sure whether CD-DPO’s gains come from the act of flipping itself or from the extra behavioral annotations. I’d like to see such baselines added.
- The conflict is manufactured, and the method needs to know what to look for
All experiments inject contamination by construction. Do real preference datasets (e.g., UltraFeedback or HH-RLHF) actually contain this kind of hidden conflict? The paper’s own calibration in Appendix A.1 doesn’t really suggest so. On top of that, CD-DPO requires labeled examples of the specific bad behavior to fit the predictor, so the behavior must be known and annotatable upfront. This sits awkwardly with the “hidden” framing in the title because in practice we often don’t know which bad behaviors are lurking in the data.
Questions for authors
-
How does CD-DPO compare to label-correcting robust DPO methods such as rDPO or RE-PO?
-
When the predictor is applied to an unmodified real dataset (e.g., UltraFeedback), what do the flipped pairs actually look like?
Dear Reviewer BSGA,
As the discussion period is nearing its end, we would like to send a gentle reminder regarding our response and the revised manuscript. If your remaining question has been resolved, we would greatly appreciate if you could consider raising your score. Please let us know if anything remains unclear.
Thank you again for your time and thoughtful feedback.