Finite-Sample Guarantees for Convex Neural Preference Learning

Pairwise preference data are commonly used to train reward models. Yet existing statistical analyses of pairwise preference learning often assume linear reward heads, leaving finite-sample guarantees for nonlinear heads less developed. We study a Bradley–Terry estimator that fits a two-layer ReLU reward head on fixed representations through a convex activation-pattern formulation. The estimator jointly fits the preference likelihood and selects among retained activation patterns using a block-sparsity penalty. Conditioning on the representations and retained patterns, we analyze squared error in reward differences on the observed comparisons. For rewards supported on at most $s$ of $P$ patterns in dimension $d$ with $n$ samples, we establish a minimax lower bound of order $s[d+\log (P/s)]/n$ under a sparse design condition. We further derive a prediction error bound of order $s(d+\log P)/n$ for the convex estimator under standard conditions. When $s$ is known, a sharper calibration of the same estimator achieves the minimax lower bound rate under a stronger restricted eigenvalue condition. Experiments with synthetic rewards, language-model representations, and five preference datasets assess reward recovery and held-out preference prediction.