acceptodds
Under review as a conference paper at ICLR 2027

Self-Configuring Hierarchies with Meta- Adaptive Hierarchical Reinforcement Learning for Multi-Agent Cooperative Control

Abstract

Hierarchical reinforcement learning organises cooperative multi-agent behaviour into levels of temporal abstraction, but the structural parameters of the hierarchy (temporal commitment intervals, per-level discount factors, and inter-level feedback gains) are almost always fixed by hand before training. We call this the selfconfiguration gap: the hierarchy cannot adapt its own structure to the task, so every new environment requires a fresh hyperparameter search, and the resulting architecture is misaligned whenever the task horizon or team size departs from the calibration setting. We introduce Meta-MOHRL, a bilevel framework in which a meta-controller maps a compact, observable task descriptor to the full structural configuration of a three-level multi-objective hierarchy once per episode; the inner loop trains the hierarchical policy under that configuration and the outer loop updates the meta-controller from observed returns across a distribution of tasks. What is meta-learned is not a policy initialisation or a set of policy weights but the temporal structure of the induced semi-Markov decision problem itself: the commitment intervals set the effective horizon each level must reason over, and the per-level discounts set the timescale at which each objective is optimised. We give a scaling argument showing that balancing estimation cost and commitment regret yields an optimal commitment interval that scales with the square root of the horizon. On a cooperative SUMO-RL traffic-signal benchmark with a heterogeneous vehicle fleet and 15 randomised network topologies, Meta-MOHRL outperforms fixed-structure hierarchical and flat multi-objective baselines in total reward and Pareto hypervolume, and recovers a per-level discount structure that was never imposed as a design choice. The meta-controller executes once per episode and adds negligible test-time overhead.

Primary area:
reinforcement learning
Keywords:
hierarchical reinforcement learning, multi-objective reinforcement learning, meta-learning, multi-agent reinforcement learning, temporal abstraction, bilevel optimization, adaptive hierarchy

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_3JKVOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper introduces a meta-RL solution for config adaptation in hierarchical cooperative MARL tasks. By experimenting on a traffic signal control task, the proposed solution shows better performance than conventional hierarchical MARL solutions.

Critical strengths
  1. The performance of the proposed method is good, outperforming previous solutions significantly.

  2. The formulation of the meta-MDP is good and practically useful. The proposed bilevel optimization addresses the MDP well.

Critical weaknesses
  1. The motivation of this paper is unclear, especially regarding the three-level design. An ablation study to show the efficacy of each level is necessary.

  2. The experiment is not sufficient. Comparing with mature baselines for the signal control task is helpful. Moreover, the experiment scope is too narrow, including only one task.

  3. The writing should be improved. For example, the notions in the introduction are not defined when they are first used.

Questions for authors
  1. What is the benefit of the proposed three-level hierarchies to conventional two-level solutions and the role-based MARL methods, where the role assignment mechanism is very similar?

  2. Can the proposed method be used in other hierarchical RL methods, or is it limited to the proposed three-level framework?

Reviewer_zg5XOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper proposes Meta-MOHRL, which uses a meta-controller to adapt temporal horizons, discount factors, and feedback gains in hierarchical MARL. Experiments on SUMO traffic networks show improvements in total reward and pareto hypervolume compared to several baselines.

Critical strengths

The idea of automatically adapting hierarchical structures is interesting and meaningful. The reported performance improvements are substantial, and the ablation studies provide useful evidence for the effectiveness of the proposed approach.

Critical weaknesses

Weakness 1: The optimization method and some theoretical claims need more clarification.

In Section 4.2, temporal horizons are obtained using Gumbel-softmax and then rounded to integers. However, it is unclear how gradients are propagated through this discrete operation. Since adapting temporal horizons is one of the main contributions, the authors should explain the actual gradient computation more clearly.

The paper also claims to learn objective-to-timescale assignments, but Table 1 already fixes each objective to a specific layer. Therefore, this claim seems somewhat overstated. The variance analysis in Appendix E also needs further justification, especially regarding its dependence on the number of agents.


Weakness 2: The experimental validation is not sufficient.

Although 15 network topologies are used, all experiments have the same horizon and number of objectives. Generalization to unseen topologies is also not tested. This makes it difficult to verify whether the meta-controller can adapt to different task conditions.

More importantly, a comparison with a well-tuned fixed three-level architecture is missing. Since the current baselines have different architectures, it is difficult to determine whether the improvement comes from meta-learning or simply from the hierarchical structure and parameter choices.

The baseline implementation also needs clarification. For example, MOSMAC is introduced in the cited work as a benchmark rather than a specific algorithm.


Minor concerns. Several figures have low readability due to small labels, legends, and insufficient visual quality (recommend to use pdf for figures). Some equations and tables also have formatting issues.

Questions for authors
  1. How are gradients computed for the discrete temporal horizons after rounding?
  2. Can the authors provide results with a well-tuned fixed three-level baseline and unseen topologies?
  3. How does the method perform with different episode horizons?
  4. Could the authors clarify the MOSMAC baseline implementation?
Reviewer_fq45OpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
2
Read review
Paper summary

This paper addresses the hierarchical  reinforcement learning setting, where N  agents optimize K competing objectives cooperatively (assuming access to a pre-defined preference simplex over objectives). The paper focuses on the hierarchical control aspect of the problem, specifically considering team control at 3 levels: strategic, multi-agent coordination, and per-agent execution. The authors observe that prior approaches often fix the task configurations of commitment intervals (i.e. option length), per-level discount factors, and hyperparameters controlling how gain is propagated from one level to the next. The proposed method, Meta-MOHRL, aims to jointly learn these task configurations via a bilevel optimization approach. In Meta-MOHRL, a task embedding is first generated from a task context vector. Given the task context vector, the meta-controller learns to output a task configuration, while the inner-level hierarchical policy learns to maximize reward given the fixed task configuration. Experimental validation on SumoRL (a cooperative traffic benchmark) reveals that Meta-MOHRL outperforms the baselines of fixed-configuration hierarchical RL and no-hierarchy baselines in terms of return and Pareto hypervolume.

Critical strengths

None

Critical weaknesses

The paper has several critical weaknesses.

First, the problem is ill-matched against the stated motivation. The paper primarily aims to address the problem of fixed task configurations within hierarchical control settings, which is a meta-learning problem. But the actual problem setting (both in the experiments and problem formulation) is hierarchical  reinforcement learning, where N  agents optimize K competing objectives cooperatively. The paper really should be addressing hierarchical RL alone, yet they've added the complexifying elements of multi-agent control and multi-objective optimization, for seemingly no reason. This obscures how well the proposed method actually addresses the problem, compared to hierarchical RL baselines which were not intended to handle multi-agent / multi-objective scenarios. It also obscures what lessons the reader can take away (currently, I cannot discern any).

Second, the writing quality does  not meet the standards of a conference publication. For one, there are serious factual inaccuracies in the  current contextualization and motivation for the problem within the Introduction. The claim that value decomposition approaches such as QMIX (Rashid et al. 2018) are hierarchical control approaches is incorrect. While the QMIX approach is hierarchical in nature as it uses hypernetworks, it is an entirely different kind of hierarchy from the high and low level control hierarchy considered by this paper. As another example, the introduction section titled "What is meta-learned, and why this is not MAML applied to HRL" contains many terms and mathematical notations that have not yet been introduced (e.g. gamma, lambda, T^1, T^2, effective horizon).

Finally, the method is presented as a general method, but the task descriptor structure (Eq 2) is task-specific. Doesn't this choice also limit the possible behaviors that can be learned, similar to a fixed task configuration for hierarchical RL? Similarly, the proposed framework also relies on manually assigning a reward objective to each level - these decisions appear somewhat arbitrary and suffer the same problem. Given the stated goals of the paper, these design decisions seem contradictory towards those goals.

Questions for authors
  1. How well does the proposed method perform compared to hierarchical RL baselines in a single-agent, single-objective scenario?
  2. How did you define task descriptor vectors for non-Sumo tasks? How does the task descriptor vector quality impact the performance of the method?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.