Same Objective, Different Optimizer: Preference-Flip Noise versus Label Smoothing in DPO

Flipping preference labels with probability $p$ and smoothing them with $\epsilon =p$ (conservative DPO, cDPO) give the same expected DPO loss but not the same optimizer. We quantify the difference of one update from a given state and show that its order in the step size depends on the update rule. The flip noise of example $i$ is exactly $(Z_i-p)J_i$, with $J_i$ the margin Jacobian, whether or not the margin is saturated. For updates affine in the gradient, such as SGD, momentum and Adam with a frozen second moment, the flip-induced change of a smooth objective is second order, $\frac{{\eta }^{2}p(1-p)}{2m^2}{\sum }_i(C{J}_i{)}^{\top }{\nabla }^{2}F(C{J}_i)$; a live adaptive second moment or an active global-norm clip can bias the expected update, giving a first-order objective difference when $\nabla F^{\top }{\beta }_B\ne 0$. Exact same-state measurements on Qwen3-0.6B, enumerating all flip patterns, confirm both orders: measured-to-predicted ratios are $0.99$–$1.06$ for SGD, and the first-order error is $0.6\%$ for live Adam at $\eta =2.5\times 10^{-6}$. A crossover step size ${\eta }^{\ast }=|a_1|/|A_2|$, computed before any measurement, predicts whether the first-order live or the second-order frozen-denominator excess is larger at twelve unseen states of Qwen3-0.6B and Qwen3-1.7B in 599 of 607 and 582 of 598 classified cases (95% intervals over state–minibatch groups $[97.1,99.8]\%$ and $[95.6,98.8]\%$), and at twelve further states built from an independent data panel in 582 of 617 and 590 of 604. An unbiased stratified estimate of the first-order coefficient replaces the enumeration, with a nominal $95\%$ interval whose coverage over 1,000 repetitions of 360 cases is 95.1% and 94.8%. In paired multi-epoch LoRA runs of Qwen3.5-2B and 4B, the flip-to-cDPO gradient-norm ratio starts near the value 1.25 that the covariance formula gives at zero margin for orthogonal Jacobians, with median 1.23, and rises to 4.7 by the third epoch. A loss audit finds that the smoothing option leaves TRL's native sigmoid loss unchanged in the newest release it runs; a code scan finds the smoothing mixture absent from that branch from one release on.