acceptodds
Under review as a conference paper at ICLR 2027

Multi-Objective Hierarchical Reinforcement Learning for Complex Multi-Domain, Multi-Timescale Systems

Abstract

Real-world decision-making problems often require integrating heterogeneous data sources, adapting to changing operating conditions, and optimizing multiple, often conflicting objectives across interacting domains and spatial and temporal scales. For these settings, we introduce Systemic Contextual Multi-Objective Hierarchical Reinforcement Learning (SC-MOHRL), a three-level hierarchical RL framework with Pareto-based multi-objective optimization. SC-MOHRL addresses problems in which objectives and decisions operate at different temporal horizons by decoupling them across hierarchical levels, while explicitly coordinating their interactions through online Pareto-based decision-making. Beyond hierarchical objective decomposition, the method introduces a unified formulation with bidirectional contextual information flow, where lower-level contexts are propagated upward to inform high-level Pareto-based decision-making across temporal scales, while high-level decisions provide top-down guidance to lower levels. The proposed method is evaluated in a complex simulated urban system with three interacting objectives: traffic flow, air quality and livability. Simulation results and ablation studies demonstrate the feasibility of the framework and the contribution of its hierarchical and contextual information sharing mechanisms.

Primary area:
reinforcement learning
Keywords:
multi-objective optimization, hierarchical reinforcement learning, bidirectional contextual information, temporal abstraction

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_MywROpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The paper proposes SC-MOHRL, a three-level hierarchical reinforcement learning framework for decision-making across multiple objectives and temporal resolutions. Low- and middle-level recurrent PPO policies optimize local scalar rewards, while a high-level Pareto Conditioned Network (PCN) learns policies conditioned on desired multi-objective returns. Higher-level actions provide context to lower-level policies, and lower-level observations and performance information inform higher-level decision-making. Training proceeds in two stages: lower-level policies are trained under random high-level actions, then frozen while the high-level policy is trained. Evaluation uses an urban transportation simulation involving vehicle control, speed-limit selection, and adjustment of car-use proportions.

Critical strengths

The application motivates coordinating decisions across different temporal resolutions. The connection between individual vehicle behavior, traffic regulation, and urban planning is intuitive, and much better than a long list of standard benchmark problems, where little can be learned. The architecture provides a concrete implementation. The component comparisons provide initial evidence that learned policies improve performance over random or fixed controllers in the evaluated setting.

Critical weaknesses
  1. The experiments do not sufficiently validate the central coordination claims. The main contributions emphasize multi-timescale coordination and bidirectional contextual information flow. However, the experiments primarily show that learned controllers outperform random or fixed alternatives. No ablation separately removes bottom-up context or top-down conditioning while preserving the remaining architecture and reward definitions. Consequently, the necessity and contribution of each information pathway remain unclear. Similarly, operating three levels at different frequencies demonstrates the feasibility of multi-timescale control, but does not establish how the proposed design addresses the objective-timescale mismatch motivating the paper.

  2. The methodological contribution needs clearer differentiation from the existing components. The framework combines PPO, PCN, contextual conditioning, and staged training. While such an integration can be valuable, the paper does not clearly establish which aspect of its coordination mechanism constitutes the main methodological advance beyond these components and the contextual MOHRL approaches discussed in Related Work. The current experiments demonstrate the behavior of the combined system, but provide limited insight into what the proposed integration specifically enables.

  3. The lower-level ablation is confounded by changing the dynamics faced by a fixed high-level policy. The high-level policy is trained with the learned lower-level controllers and then evaluated after those controllers are replaced by random or fixed alternatives. Performance degradation may therefore reflect a mismatch with the dynamics encountered during training, rather than isolate the benefit of the proposed lower-level design.

Questions for authors
  1. What are the exact high-level policy inputs, and how are lower-level rewards accumulated and discounted between successive high-level decisions?

  2. Could the authors provide further evidence or analysis to substantiate the claimed benefits of bidirectional contextual information flow? In particular, how can their contributions be distinguished from the general benefits of using learned controllers?

  3. What evidence demonstrates that the proposed coordination addresses objective-timescale mismatch? How sensitive are the results to the chosen decision intervals?

1 comment
AuthorsOpenReview ↗

Reply to the reviewer

Thank you for your insightful and valuable reviews. Following are the answers to your questions:

  1. High-level inputs were described in Section 3.4. In the current simulation, lower-level reward didn’t accumulate during high-level decisions, high-level multi-objective reward vector is formulated by combining information gathered from the low and middle levels at the respective time step, see equation (27) in Appendix A.
  2. We are currently working on the ablation to evaluate if the contextual information is actually improving the optimization at PPO. However, due to the substantial training time required, the results are not available within the rebuttal period.
  3. In this work, we aim to develop a RL-based framework to address a complex real-world problem with different hierarchical decision levels. The different temporal horizons for different hierarchical levels come from their real-world temporal horizons. Our framework is aimed to optimize RL to address the real-world problems and not to optimize real-world to address RL optimization.
Reviewer_UgbqOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes SC-MOHRL, a three-level hierarchical reinforcement learning framework for optimizing conflicting objectives across different temporal scales. It combines PPO-based lower-level policies with a high-level Pareto Conditioned Network (PCN) and introduces bidirectional contextual information sharing to coordinate decisions across hierarchical levels. Experiments in an urban traffic simulation demonstrate improvements in Pareto coverage and preference-tracking accuracy compared with random and fixed-policy baselines.

Critical strengths

The story is interesting, as it brings together multi-objective optimization and hierarchical reinforcement learning to handle conflicting objectives across different timescales.

Critical weaknesses
  1. I cant see any complex problem in the motivation. "Real-world applications often require integrating heterogeneous data sources (e.g. behavioral and environmental signals), operating under non-stationarity and partial observability, and optimizing multiple, often conflicting objectives across interacting domains and spatial and temporal scales." It is unclear what makes the problem fundamentally challenging enough to establish a clear research gap.
  2. LSTM and PPO rather outdated. Both are well-established techniques and simply putting them together in a RL framework feels like a solution from a decade ago. I struggle to see what is technically new here.
Questions for authors
  1. I find the baseline settings in Section 4.3 rather arbitrary. Why use driving shares of 10%, 50%, 90%? Why choose speeds of 30, 50, 70 km/h? I don't see any clear justification for these values. Without a sensitivity analysis, it's hard to tell whether the improvements actually come from the proposed method rather than weak baselines.
  2. The two-stage training seems unnecessary. The authors claim that freezing the lower-level policies improves training stability. However there is no comparison with joint training or alternating training. Joint fine-tuning is also missing. More importantly freezing the lower-level policies seems to reduce bidirectional coordination to information sharing rather than actual joint policy adaptation. The motivation for this design remains unclear.
1 comment
AuthorsOpenReview ↗

Reply to the reviewer

Thank you for the insightful and valuable reviews. Following are the answers to your questions:

  1. Settings for the driving share and speeds are based on the realistic situations emerging from our simulated traffic environment. Thereby speeds were set to be 30, 50, 70 km/h, and the driving shares were the minimum, maximum, and average values achievable by our simulations i.e. 10%, 50%, 90%.
  2. We designed the two-stage training procedure to reduce the training complexity, making the lower layer controllers face most possible states beforehand will make the higher layer training more stable. We base our evaluation on prior scientific work [1]. We have also added the citation to the paper. Freezing the lower-level policies do not reduce bidirectional coordination to information sharing, because during training process, the random high-level information is also given to the lower levels which leads the lower-level policies trained with different inputs.

[1] Hutsebaut-Buysse, Matthias, Kevin Mets, and Steven Latré. "Hierarchical reinforcement learning: A survey and open research challenges." Machine Learning and Knowledge Extraction 4.1 (2022): 172-221.

Reviewer_1PmhOpenReview ↗
Rating
1
Confidence
4
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper investigates conflicting objective coordination in multi-domain, multi-timescale systems, proposing Systemic Contextual Multi-Objective Hierarchical Reinforcement Learning (SC-MOHRL), which combines a three-layer decision architecture with low- and mid-level control and a high-level Pareto tradeoff. The low- and mid-level layers employ recurrent PPO to control vehicle acceleration/deceleration and speed limit, respectively; the high-level layer uses a Pareto Conditioned Network (PCN) to adjust the driving ratio. The low- and mid-level layers are first jointly trained and frozen under random high-level instructions, and then the high-level PCN is trained; each layer operates at intervals of 10 seconds, 10 minutes, and 50 minutes, exchanging bottom-up observations and top-down action contexts. The core contributions lie in the integration of three-layer temporal abstraction, cross-layer context exchange, and existing Pareto conditional learning. The experiment demonstrated feasibility using a single SUMO city simulation and 10 homogeneous agents; the actual direct optimization targets are traffic efficiency and air quality, while livability is derived from air quality and fixed population characteristics, and is not an independent third optimization direction.

Critical strengths
  1. The problem motivation is clear and valuable for research: Section 1 points out that different decision-making authorities and time scales influence each other, and local optimization at a single level may not be able to handle the trade-offs of the entire system, which is a problem worthy of study.

  2. Hierarchical roles and decision frequencies are specific: Section 4 clearly outlines the roles of urban planners, traffic managers, and road users, as well as the three-level decision intervals. Compared to merely abstractly claiming the existence of hierarchy, this paper provides an understandable application correspondence.

  3. Phased training has a reasonable motivation for stability: Section 3.5 trains and freezes the lower layer first, then trains the higher layer, making the lower layer controllers faced by the higher layer more stable; although this scheme has not been proven to be optimal, the design motivation is reasonable.

  4. The higher layer retains multiple trade-off solutions: Section 3.4 uses PCN to select behavior based on the objective reward vector, rather than providing only a single fixed weighted solution for the higher layer; Figure 3 presents the achievement under different objective rewards.

  5. Positive results were observed in the tested ablation dataset: the hypervolume in Table 1 was 22.46, higher than the 19.43 for the randomized high-level ablation dataset, representing an improvement of approximately 15.6%; the mean target tracking error was 0.7165, lower than the listed randomized and fixed-rate lower-level versions. This supports preliminary feasibility but does not yet indicate superiority over a strong baseline.

  6. Some limitations were clearly disclosed: Section 4 acknowledges that habitability is not an independent objective; Section 4.3 also states that the complete method did not achieve the lowest error for the last two of the five tested objectives. These limitations help readers correctly interpret the results.

Critical weaknesses
  1. Core contributions have not been independently validated experimentally and a competitive baseline is lacking. This paper's core contributions are multi-timescale Pareto coordination and bidirectional contextual information flow. However, existing ablation methods primarily replace the controller with random or fixed behavior, failing to remove bottom-up, top-down, or time-based contexts, and not comparing single-layer, two-layer, and same-frequency versions. Therefore, the benefits cannot be attributed to the claimed mechanism. Section 4 omits comparisons due to the need for additional design to modify existing methods, and this is insufficient to demonstrate superiority over existing MORL/HRL methods (see Abstract; Sections 1, 2, 3.2, 4.3; Table 1).

  2. The feedback aggregation and discounting across timescales lack a clear and consistent objective. Decisions are made every 300 basic steps at the high level, but Equation (26) only states the average low-level feedback and air quality feedback at that moment, without specifying whether it is a single snapshot, the accumulation of the entire 50-minute interval, or an interval average. If only the boundary is sampled, stops and pollution within the interval may be missed. Equations (1) and (11) also do not clarify the time unit for discounting; Section 4.1 sets both low- and mid-level discounts to 0.99, with physical timescales of 10 seconds and 10 minutes respectively, which are not equivalent. Different discounts can be reasonably designed, but they must be related to the overall system objectives.

  3. The dimensions of air quality feedback and the simulation model lack sufficient definition. Eq. (23) directly takes the minimum value of the threshold difference between PM2.5, black carbon, and total particulate concentration, but particulate number concentration and mass concentration have different dimensions. The text does not explain that they should be converted into comparable dimensionless quantities first. Eq. (22) adopts linear or squared forms based on positive and negative values, allowing the numerical scale to directly affect the feedback. On the other hand, f_AQ only maps the spatial pollution field using the number of vehicles and average speed, without providing a model, coefficients, calibration errors, or test range, making it difficult to confirm the practical significance of the air pollution tradeoff.

  4. Frozen and locally single-target lower layers limit the Pareto set accessible to higher layers. The lower layer is trained with fixed local feedback under random higher-level actions, and then completely frozen; the middle-level feedback r_AQ−p_traf itself has a fixed exchange method between air quality and traffic penalties. Therefore, the higher-level PCN can only search within the behaviors achievable by this frozen controller, and cannot guarantee coverage of all the important trade-offs of the original system. The principle of "not adopting fixed feedback pure quantization" in this paper should be limited to the higher level and not directly extended to the overall architecture.

  5. Contextual Information Flow and the Executable Algorithm of PCN Still Have Key Gaps. The high-level context of Equation (6) only includes low- and mid-level observations, but F in Equation (26) outputs low-level feedback; the feedback in Equation (21) requires the distance to the next time step, so it is currently impossible to see how to causally calculate it from the listed current observations. The inputs to the high-level policy only list o_h, target reward, and remaining field of view, without explicitly listing c_h. The downward transmission is a proportional increment action rather than a target proportion or preference; its duration and its effect on the mid-level speed limit are also incomplete. Equation (7) is empty in the strictly earlier time set at t=0.

  6. Evaluation biased towards self-target tracking and lack of statistical evidence for training layers. Taking five target points from the foreground obtained in this paper and then replacing the lower layers with the same high-level strategy for testing will simultaneously introduce target selection bias and changes in training distribution. This result can test the sensitivity of the frozen high-level to the replacement of the lower layer, but cannot prove that the best performance after retraining the replacement system is worse. 50 Monte Carlo evaluations are not equivalent to multiple independent training seeds; the hypervolume in the table does not have a confidence interval, and the aggregation method of the error standard deviation is unclear.

  7. The Three-Objective Description Goes Beyond Actual Direct Optimization. The abstract and introduction emphasize three interacting objectives: traffic, air quality, and livability, but PCN only provides two-dimensional feedback. With fixed population characteristics, livability is merely a function of PM2.5; if its direction aligns with the air quality index, it does not add an independent trade-off direction. Although Section 4 acknowledges this limitation, it does not present actual assessment results of livability or new conflicts, therefore, it cannot currently support the capability of three-objective optimization. Please position this paper as an optimization of two objectives plus a derived livability index, providing the index value and monotonicity conditions for PM2.5. To maintain the claim of three conflicting objectives, independently modifiable accessibility, equity, travel costs, or noise indices should be added, and a new three-dimensional frontier assessment should be conducted.

  8. A single small-scale simulation and direct control of vehicle proportions are insufficient to support a broad system claim. The experiment only describes 10 homogeneous agents, a single urban environment, and a fixed time scale, without providing the road network scale and background traffic flow, making it difficult to assess the claimed complexity. The direct adjustment of driving proportions every 50 minutes by upper levels does not explain whether in-process vehicles are removed, how to conserve total trips, alternative vehicle capacity, and switching costs. If reducing traffic flow improves air quality without considering the costs to other passengers, the observed trade-offs may be dominated by simulation simplification.

Questions for authors
  1. Under the same observation, action, sample, and computational budget, incorporate directional contextual ablation, remove LSTM/previous-step action feedback, and two-layer and same-frequency versions; at least compare standard PCN, preference-conditional PPO weighted scanning on the same high-level induced environment, and Envelope Q-learning applicable to discrete actions. Further verification should be conducted using a small-scale solvable environment to determine whether the timescale mismatch truly caused difficulties ( See [1], [3], and [6] for relevant literature).

  2. Define common goals using a base clock, and clearly list the intervals, causal indices, termination processes, and discounts for each layer of feedback. If pursuing the same time preference, R_h(t)=Σ_{k=0}^{299} γ_base^k r_vector(t+k) can be used, and γ_base^300 can be used as the discount for adjacent high-level steps; if different target perspectives are deliberately adopted, the reasons should be explained and sensitivity analysis should be performed. Undiscounted full-process entity indicators should also be added.

  3. Publicly disclose the pollutant units and thresholds, first normalize them using their respective positive scales, for example (q_j−AQ_j)/s_j, then aggregate them, and explain the design of the negative squared values. Provide the equation or surrogate model for f_AQ, PALM data sources, spatial grids, and retained test errors; verify sensitivity under different traffic flow, speed, and weather conditions, and list the applicability range of the surrogate model as a limitation.

  4. Clearly define the set of achievable rewards induced by the frozen lower layer, and call the result the empirically non-dominated solution within this restricted set. Compare preference-conditional lower layers, multi-skill libraries, or phased joint fine-tuning to see if the frontier can be expanded; compare the frozen strategy with alternative training schemes using the same training budget. Do not directly equate the action instruction context with the complete Pareto preference passed down.

  5. Provide stepwise pseudocode and input tables for each level, clearly distinguishing the order of pre-action observations, interval rewards, action updates, and context readings; explicitly list whether c_h is input to the PCN or only used to form feedback. Define the transmission methods for the initial value at t=0, the cumulative driving proportion, and the mid-level speed limit, as well as the recursive update, sampling, and data retention rules for the PCN target reward. The standard PCN already has conditionalized rewards and updated interaction data; the changes in this paper should be explained [6].

  6. Retain the replacement test for dependency analysis, and retrain the high-level for each replacement lower layer. Compare all methods with a common and independent set of objectives, the same initial state, and an evaluation budget; separate target tracking error from leading-edge quality, and report the normalized hypervolume, coverage or utility metrics, and the confidence interval across training seeds. Disclose the original reward vector, the selection criteria for the reference point (2,11), and the calculation formula.

  7. Clarify the relationship between the 10 agents and the total traffic demand; fix the total travel demand, explain whether the conversion only affects new trips or also affects trips in progress, and include the completion rate, time and capacity constraints of non-motor users. Increase stress testing of traffic demand, agent heterogeneity and decision frequency, and verify at least on a second reproducible road network or synthetic multi-scale benchmark. MoST can be used as a multi-modal data source, but air quality and control interfaces still need to be constructed separately [5].

  8. Eq. (1) in Section 3.1 and Eq. (11) in Section 3.4: Clarify the relationship between the definition of indefinite duration and the finite 20-hour experiment; provide processing for high-level discount values, remaining vision, and terminal rewards.

  9. Section 3.1, page 5: The statement that "expected cumulative reward cannot be maximized in multi-objective scenarios" is too broad. If utility/preference is known, a corresponding optimal solution can still be defined; it should be explained that this paper adopts a setting where preferences are unknown and multiple solutions are retained.

  10. Sections 3.3 and 4.1: Supplement the LSTM dimension, hidden state reset, rollout and truncation length, GAE, clip, batch, optimizer, parameter sharing method, and rate limit execution constraint for recurrent PPO.

  11. Sections 3.4 and 4.2: The units for "100 transitions" and "20 Pareto points per selection" are unclear. The number of trajectories, transitions, and reward points should be distinguished, and the total number of high-level training episodes, warm-up times, and update times should be listed.

  12. Section 5: Distinguish between offline RL and off-policy/playback supervised training. Sections 3.4 and 4.2 describe continuous interaction and data updates, which are different from offline RL with a fixed dataset; online decision-making and online learning should also be distinguished.

  13. Page 9 AI Use Statement: The first part states that no AI implementation methods were used, while the second part states that AI was used to build and modify programs; the scope of assistance and wording should be clarified to avoid contradictory statements.

1 comment
AuthorsOpenReview ↗

Reply to the reviewer

We sincerely appreciate all these valuable and insightful comments which are helpful in further improving our paper. Followings are the answers to your questions:

  1. We have done more relevant ablation studies based on your suggestions, but unfortunately, the training takes a very long time, and we cannot get the results before the rebuttal time ends. Also, a direct comparison with standard PCN or preference-conditional PPO is not straightforward due to fundamental differences in the current frameworks. Our simulation has complex interactions among different hierarchical levels, making it difficult to construct a simplified and comparable small-scale environment. Unfortunately, the references you mentioned in your review were missing. We would appreciate it if you could add them so that we can go through these relevant literatures.
  2. In the simulations, we used different temporal preferences for different hierarchical levels because of their real-world decision-making responsibilities. We set the discount factor for the high level 1 because its responsibility is to focus on long-term decisions. Details about the simulations including intervals and discount factors for different levels have been added to the revised version (Table 3 in Appendix B).
  3. We have added the pollutant units and thresholds to Appendix A. The details of how to build the large-scale AQ model providing data to our setup and its sensitivity analysis are out of this paper’s scope but are widely published at atmospheric science forums eg [1][2]. Applicability range of the surrogate model is included in the revised paper.
  4. We noticed that freezing the lower-level policy may restrict the achievable Pareto frontier. Our goal in the paper is to investigate hierarchical multi-objective RL with a frozen lower-level policy, rather than to claim global Pareto optimality. Current design is intended to reduce the complexity of joint training across three RL levels with different decision timescales.
  5. Pseudocode, initial values of the experiments and other details have been added in the revised version in Appendix B and C.
  6. Evaluation of “replaced the trained lower-level policies with fix or random actions while kept the fixed trained high-level policy” in ablation study is removed. We agree it is a mismatch to replace the lower-level policies with random or fixed alternatives and evaluate the trained high-level policy under these. We are retraining the high-level policy for each replacement, but it is not complete at the moment. We also consider adopting more appropriate evaluation metrics. The original reward vectors were normalized before evaluation. The reference point (2,11) was selected because it is dominated by all evaluated solutions.
  7. We have explained the traffic demand in the revised version in Appendix B. Testing in another road network is not realistic at the moment, as the AQ function is built on available atmospheric simulation results in the selected area.
  8. In our experiments, the episode duration is fixed at 20 hours; each episode terminates automatically when the time limit is reached, without requiring any triggered termination condition. We set the high-level discount value to 1. There are no additional terminal rewards for high level.
  9. We has revised the Section 3.1 to: “…finding a Pareto front describing the trade-offs between conflicting objectives and providing a set of non-dominated policies is required when the preferences are unknown.”. 10 and 11. Simulation and training details are added in Table 2 and 3 in Appendix B.
  10. We have removed the description of the offline RL part in Section 5.
  11. We strictly follow ICLR rules of AI use and report according to the templates and website https://iclr.cc/Conferences/2027/AIPolicyForAuthors. “Implement methods” and “create or edit software code” are clearly in two different categories. However, it is true that “create or edit software code” belongs to “implement methods”. We have revised this part.

[1] N. Atashi et al., “Quantifying the impact of vehicle fleet electrification on local aerosol concentrations in Helsinki using high-resolution Large Eddy Simulation,” EGUsphere, preprint, 2026, doi: 10.5194/egusphere-2026-341.

[2] Du, Yaxing, et al. "Large-eddy simulation of aerosol concentrations in a realistic urban environment: Model validation and transport mechanism." Environmental Pollution 358 (2024): 124475.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.