Conservative Dueling Bandits: Preference Learning under Baseline Constraints

Contextual dueling bandits learn from pairwise comparisons, as in fine-tuning language models from human preferences. Standard algorithms explore aggressively, however, and may show users two poor options. We require that, at all times, the cumulative score of the better displayed item stay above a $(1-\alpha )$ fraction of a trusted baseline's. An item's score is its probability of beating the best available item, so the constraint needs no absolute rewards and is invariant to utility shifts. We show that directly transferring the safety test of the conservative bandit algorithm CLUCB can incur linear regret. For linear utilities, MaxInPwBC certifies safety through utility gaps and falls back to comparisons that pair the baseline with an informative partner. It is safe with high probability at all times, with regret $\tilde{\mathcal{O}}(d\sqrt{T}/\kappa )$ plus a horizon-independent cost of conservatism. For general utility classes, BTIGWBC builds safety certificates from an online logistic-regression oracle, achieving anytime safety, sublinear regret, and finitely many fallbacks. Both algorithms need only bounds on baseline scores. In experiments, both satisfy the constraint, which their unconstrained counterparts often violate, at a moderate cost in regret.