Multi-Objective Constraint Inference using Inverse reinforcement learning

Constraint inference from expert demonstrations is widely considered essential to align reinforcement learning agents with safety boundaries and operational guidelines. However, existing approaches typically assume homogeneous demonstrations, generated by a single expert or multiple experts with identical objectives. They also have limited ability in capturing individual preferences and often suffer from computational inefficiencies. In this paper, we introduce Multi-Objective Constraint Inference (MOCI), a novel framework to jointly extract shared constraints and individual preferences from heterogeneous expert trajectories, where multiple experts pursue different objectives while adhering to the same safety constraints. MOCI effectively models and learns from diverse, and potentially conflicting, behaviors. Empirical evaluations demonstrate that MOCI improved predictive performance, showed competitive computational efficiency, and recovered clinically credible preferences and constraints in a longitudinal lung cancer treatment setting. These results establish MOCI as an accurate, flexible, and practical approach to the real-world constraint inference and preference learning tasks in inverse reinforcement learning with heterogeneous experts.