Preference-Guided Multi-Objective Reinforcement Learning with Explicit Objective Bounds

Preference-based multi-objective reinforcement learning (MORL) learns policies that adapt to user-specified preferences, but practical decision problems often involve explicit objective bounds as well. Existing preference-based MORL does not directly handle such bounds, while constrained MORL does not explicitly preserve a given user preference. Motivated by this gap, we formulate preference-guided MORL with explicit objective bounds. In this setting, preference alignment, Pareto-front progress, and bound satisfaction can induce conflicting local updates, while the realized RL update may further drift from the selected direction during optimization. To address these challenges, we propose PBO, a single-projection method for preference-guided MORL with explicit objective bounds. Specifically, PBO constructs a preference-guided gradient-coefficient reference and obtains the final policy direction through a single projection enforcing preference-preservation and bound-handling constraints in the induced Fisher geometry. A directional KL guard further limits the accumulation of drift during repeated policy updates. We also provide first-order bound guarantees and a conditional contraction bound for constraint violations. Across control benchmarks and offline recommendation, PBO consistently achieves strong performance in preference following, bound satisfaction, and feasible Pareto-front quality.