Preference-Grounded Latent Control: Joint Representation and Policy Learning

Learned latent states underpin world-model agents, embodied robots, and recurrent controllers by determining which information is available for control. Yet conventional latent control optimizes environment prediction or task performance, which need not reflect user preferences. Preference optimization supplies the missing alignment signal, enabling latent states to retain user-relevant distinctions and actors to select preference-consistent actions in otherwise similar situations. Direct Preference Optimization (DPO) is especially attractive because it learns directly from pairwise comparisons without a separate reward model or nested reinforcement-learning procedure. However, standard DPO assumes a shared observable context, whereas latent sequential control must propagate trajectory preferences through an internal state process. Its usual likelihood-ratio reduction therefore cannot jointly train the latent decision state and actor. To address this, we introduce preference sufficiency, which characterizes whether a latent state preserves every distinction required for preference-aligned control, and a representation-error measure for information lost through latent aliasing. We then propose Observation-Anchored Latent Soft-Advantage DPO, a closed-form direct preference framework that anchors the reference policy in observable history space while jointly learning the latent decision state and actor. We prove that the resulting soft-advantage score is canonical and Bellman-consistent, derive an exact actor–representation error decomposition, and show that its representation term vanishes if and only if the state is preference-sufficient. Experiments across sequential-control tasks validate the effectiveness of jointly learning preference-relevant representations and policies.