Multi-Objective Hierarchical Reinforcement Learning for Complex Multi-Domain, Multi-Timescale Systems

Real-world decision-making problems often require integrating heterogeneous data sources, adapting to changing operating conditions, and optimizing multiple, often conflicting objectives across interacting domains and spatial and temporal scales. For these settings, we introduce Systemic Contextual Multi-Objective Hierarchical Reinforcement Learning (SC-MOHRL), a three-level hierarchical RL framework with Pareto-based multi-objective optimization. SC-MOHRL addresses problems in which objectives and decisions operate at different temporal horizons by decoupling them across hierarchical levels, while explicitly coordinating their interactions through online Pareto-based decision-making. Beyond hierarchical objective decomposition, the method introduces a unified formulation with bidirectional contextual information flow, where lower-level contexts are propagated upward to inform high-level Pareto-based decision-making across temporal scales, while high-level decisions provide top-down guidance to lower levels. The proposed method is evaluated in a complex simulated urban system with three interacting objectives: traffic flow, air quality and livability. Simulation results and ablation studies demonstrate the feasibility of the framework and the contribution of its hierarchical and contextual information sharing mechanisms.