- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
The paper proposes SC-MOHRL, a three-level hierarchical reinforcement learning framework for decision-making across multiple objectives and temporal resolutions. Low- and middle-level recurrent PPO policies optimize local scalar rewards, while a high-level Pareto Conditioned Network (PCN) learns policies conditioned on desired multi-objective returns. Higher-level actions provide context to lower-level policies, and lower-level observations and performance information inform higher-level decision-making. Training proceeds in two stages: lower-level policies are trained under random high-level actions, then frozen while the high-level policy is trained. Evaluation uses an urban transportation simulation involving vehicle control, speed-limit selection, and adjustment of car-use proportions.
Critical strengths
The application motivates coordinating decisions across different temporal resolutions. The connection between individual vehicle behavior, traffic regulation, and urban planning is intuitive, and much better than a long list of standard benchmark problems, where little can be learned. The architecture provides a concrete implementation. The component comparisons provide initial evidence that learned policies improve performance over random or fixed controllers in the evaluated setting.
Critical weaknesses
-
The experiments do not sufficiently validate the central coordination claims. The main contributions emphasize multi-timescale coordination and bidirectional contextual information flow. However, the experiments primarily show that learned controllers outperform random or fixed alternatives. No ablation separately removes bottom-up context or top-down conditioning while preserving the remaining architecture and reward definitions. Consequently, the necessity and contribution of each information pathway remain unclear. Similarly, operating three levels at different frequencies demonstrates the feasibility of multi-timescale control, but does not establish how the proposed design addresses the objective-timescale mismatch motivating the paper.
-
The methodological contribution needs clearer differentiation from the existing components. The framework combines PPO, PCN, contextual conditioning, and staged training. While such an integration can be valuable, the paper does not clearly establish which aspect of its coordination mechanism constitutes the main methodological advance beyond these components and the contextual MOHRL approaches discussed in Related Work. The current experiments demonstrate the behavior of the combined system, but provide limited insight into what the proposed integration specifically enables.
-
The lower-level ablation is confounded by changing the dynamics faced by a fixed high-level policy. The high-level policy is trained with the learned lower-level controllers and then evaluated after those controllers are replaced by random or fixed alternatives. Performance degradation may therefore reflect a mismatch with the dynamics encountered during training, rather than isolate the benefit of the proposed lower-level design.
Questions for authors
-
What are the exact high-level policy inputs, and how are lower-level rewards accumulated and discounted between successive high-level decisions?
-
Could the authors provide further evidence or analysis to substantiate the claimed benefits of bidirectional contextual information flow? In particular, how can their contributions be distinguished from the general benefits of using learned controllers?
-
What evidence demonstrates that the proposed coordination addresses objective-timescale mismatch? How sensitive are the results to the chosen decision intervals?
Reply to the reviewer
Thank you for your insightful and valuable reviews. Following are the answers to your questions: