Conformal Preference Weighting for Label-Noise-Robust Direct Preference Optimization

Direct Preference Optimization (DPO) and its uncertainty-aware variants assume the preference label attached to each response pair is correct. A flipped label does not merely add noise: it inverts the training signal, so the policy is trained toward the response it should reject. TUR-DPO addresses this by reweighting pairs with a model-internal uncertainty heuristic. We show that heuristic is non-discriminative in practice — across every run we measure it emits a single constant ($0.8912$, zero variance) for every pair, clean and corrupted alike — and, on a complete grid of four arms, four noise levels and three matched seeds, that it is worse than not reweighting at all, falling below plain DPO in $8$ of $9$ paired runs under noise. We replace it with Conformal Preference Weighting (CPW), which derives the weight from a held-out calibration split via split conformal prediction and conformal risk control, and with CPW+, which fuses a local and a relational view of each pair before a single selective-risk calibration. CPW+ exceeds DPO under label noise by $+0.023$ paired accuracy over the runs that certified a threshold ($6$ of $7$ positive), and on accuracy retention — each arm against its own clean baseline — exceeds TUR-DPO in all nine matched runs at every noise level. Calibration uses no ground-truth corruption labels at any point. We are equally explicit about what does not work: the method costs $0.005$ accuracy on clean data; a third, optimization-aware view of our own design scores flipped pairs higher than clean ones and is given zero weight; the calibration certified no threshold in $2$ of $12$ runs; the structural risk term was degenerate under the extractor used for the accuracy runs, so those numbers are a lower bound rather than an estimate; and on MATH the weights fail to separate corrupted pairs (AUROC $0.605$) and CPW is the weakest arm.