Test-Time Personalization via DPO with Latent Reward Features

Standard preference alignment of language models (LMs) relies on a single reward model (RM), overlooking the diversity of human preferences. While recent work on personalized RMs address this by learning reward features—representing shared latent preference dimensions—together with user-specific weights, updating the alignment objective to these personalized weights may still require repeatedly performing costly preference optimization for each user. In this paper, we propose TPD (Test-time Personalized Decoding), a framework for on-the-fly personalization of the LM that avoids model retraining for each user. TPD first learns a set of shared feature-level policies through a DPO-like objective, bypassing the explicit learning of reward features. At test-time, TPD then balances these policy logits using weights adapted to the user's preferences, following a multi-objective decoding approach. Importantly, we show that TPD naturally extends to a broad class of alignment objectives based on $f$-divergences. Experiments on synthetic and real-world datasets demonstrate effective steering of generation toward maximizing individual user satisfaction.