acceptodds
Under review as a conference paper at ICLR 2027

Hierarchical W-Learning

Abstract

Inspired by a model of the brain called projective simulation, which has attracted interest within the physics community in recent years, we develop a simple and general framework for hierarchical reinforcement learning. The proposed method extends the conventional action-value Q-function to a hierarchical W-function, enabling an agent to select actions according to a hierarchy of strategies. We first present a rigorous formulation of the hierarchical framework, together with the corresponding W-learning algorithm and the hierarchical policy gradient theorem. We then demonstrate the proposed method on a navigation task to illustrate the learning procedure and its underlying mechanism. Furthermore, we extend the hierarchical framework to several modern reinforcement learning algorithms, including hierarchical deterministic policy gradient (DPG), advantage actor-critic (A2C), proximal policy optimization (PPO), and soft actor-critic (SAC), and evaluate them on the continuous navigation task along with MuJoCo control benchmarks. Experimental results show that incorporating an appropriate hierarchical strategy can consistently improve learning performance over conventional reinforcement learning methods, provided that the hierarchy is well designed and the update parameters are properly chosen.

Primary area:
reinforcement learning
Keywords:
Reinforcement learning, hierarchical, Q-learning

Reviews

Among:
Shape:
1234this paper, Bottom 25%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.75 over 4 reviews, above 25% of the others.
Reviewer_UWjfOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper present a novel hierarchical reinforcement learning algorithm inspired by some concept from projective simulation. The authors propose a framework that generalize flat action-value function to new hierarchical W-function, letting the agent behave accordingly to an hierarchical strategy ω\omega. The authors present an off-policy learning algorithm derived from Q learning and the policy gradient, and actor-critic, case by deriving the corresponding policy gradient theorem. Finally, the authors propose a deterministic PG extension of the algorithm and provide experimental results on continuous and discrete navigation tasks.

Critical strengths
  • The method presented can theoretically be put on top on various existing algorithms. The authors show this strength by combining their framework with REINFORCE, DPG, A2C, PPO and SAC.
Critical weaknesses
  • The presentation of the approach is unclear and confusing in several parts. The authors adopt concepts from other literature without properly explaining them. For example, the introduction presents the “clip” as a core part of the framework, but the concept disappears in the following sections.
  • The setting relies strongly on prior knowledge of the task, which is quite a restrictive assumption.
  • The Authors state that "[...] incorporating an appropriate hierarchical strategy can consistently improve learning performance [...]", which is not shown in any part of the work. Results from Figure 6 shows the same performance regardless of using the hierarchical approach, with the shaded areas (which I assumed to be confidence intervals) always overlaps. Figure 3 and 4 are consistent with what I said as the W-learning approach is statistically equivalent for every case. The only case where the hierarchical approach is slightly better is in Fig. 3a, but at the cost of having a tuned hierarchical structure defined a priori.
  • The theoretical analysis is mostly known result presented in a very confuse way and none of the assumptions they rely on are stated or cited. Thm. 3 proves convergence to a W∗W^* that was not characterized by the Authors.
Questions for authors
  • Can the Authors share more details on the implementation of the hierarchical version of the baselines (PPO, A2C, SAC)?
  • Can the Authors share more details about the definition of gg? It is presented as a core parameter, but it is still unclear to me how it is determined.
  • Regarding the continuous experimental section, what are the hyper parameters and g function of all methods? Only the inverted pendulum ones are shown.
  • How are the hierarchy and the strategy ω\omega defined in practice? No details have been shared on this part.
  • Generally, can the Authors define clearly the framework? A lot of parts remain unclear to me. What is a strategy and how should be defined? How does the policy function work given a set of parameters?
3 comments

Clarification:

  • “Clip’’ is the terminology used to discuss the contexts related to the projective simulation model while in the main part of the paper it is equivalent to the word “node” in the graph.

  • For the plot issue, sometimes we put many curves in one figure. It is better to compare green and gray, red and blue curves in figure 3a 3b, and compare gray curves with other three in fig. 3c. Also, for figure4, it is better to compare green and gray, red and blue curves in figure 4a, and compare the gray curve with other two in fig4.b, fig4,c. In figure 6a and figure 6b, it is better to compare the blue/red curves with the green/gray curves. For MuJoCo, we conducted four more representative experiments in appendix H to highlight the differences. We did average on all samples rather than some of the best trials, so the overlaps between confidence intervals sometimes are big.

  • The convergence proof is the standard textbook approach as we have cited in the paper, applied to our new defined W function with additional hierarchical structure.

For questions:

  • Implementations of the HSAC and HPPO, together with some theoretical constructions, have been added in appendix B4 and appendix B5.

  • The defintion of the paramter gg deponds on the task. For tasks with sparse reward, we can set intermediate rewards to encourage the exploration, for example the difference between current state and next state. For continuous control tasks, one can construct functions which are maximal when the state is near stable position, or with small velocity, and set them as the parameter g. It is also useful to look at the reward designed by the benchmark.

  • There are three methods over four MuJoCo experiments so we have 12 sets of parameters for gg. All of them take the form

g=R×C×(A+Bx)\qquad\qquad g=R \times C\times (A+Bx)     if we want a bigger x, for example velocity in x direction for Walker2d or Hopper, or g=R×CA+Bx\qquad \qquad g=R\times \frac{C}{A+Bx}      if one requires a small x, for exmaple, small actions, angles, angular velocities.

     R is the reward and A, B,C are constants to be determined.

  • In our framework, we provide two ways to develop the structure ω\omega. One is building the structure from smaller one which has been pre-trained in easier situation, the other is cutting off legs from a bigger structure. For example, in the navigation task, one can first train the agent when there is no obstacle and add higher-level strategies when the agent is put into the environment with obstacles. Or we can directly start from a fully connected high-level structure and then cut off the legs with low transition probability but keep all actions connected.

Thank you for your responses and the revision on the manuscript. However, I am still not convinced by the approach.

As the plot of Fig. 3, 4 (in 6 only the HPPO show some improvement w.r.t. its flat version, the other methods are identical) might suggest some type of improvement, this is mostly due to a specific optimization based on prior knowledge (g parameter) and not on the hierarchical structure. In Fig. 24 this phenomena can be seen clearly as all the results with g=rg = r perform equally or worse w.r.t. the flat version (HA2C is still not defined as an algorithm). This makes me raise some concerns about the usefulness of the approach.

Moreover, as stated before the theory presented is an application of standard results and not a novel results brought by the Authors. I also have some concerns on the Proof of Thm. 3 as the operator H in Appendix A uses gg instead of the reward. Its fixed point W∗W^* is therefore the value of the current policy under gg, not the optimal value of the task. Can you elaborate on this to help me understand better your proof?

Finally, can you elaborate on this statement done in section 4 "It enables us to find stronger policy compared to the globally optimal policy by constructing more complicated clip structure."? How can you find a policy stronger than the optimal one?

Overall, because of the rigidity of the approach, which forces the practitioner to hand-design several parts of the algorithm, in addition to what was already said, I keep my score.

  • Good question. One fact we should note that different strategies may lead to the same action, therefore the strategy is not unique given the optimal policy. The parameters gig_i introduced here are used to uniquely determine the strategy. In practice, we set the first level g1=rg_1=r or the last level gl=rg_l=r in order to get the optimal value of the task. But that rule is not rigid. Exploring how to determine the value of gig_i so that the agent could perform better is one of the central goal of our work. The proof holds for all bounded gi<∞g_i < \infty, because it will be cancelled when we calculate HW−HWˉHW-H\bar{W}.

  • Suppose we have two strategies. I is following the standard RL policy and the other II is to do something else following other structure. During the training, if the agent finds it will get more reward following I, then it will choose the standard policy ohterwise the agnet will choose strategy II. So we can see, the learned policy is a least as strong as the standard one because that is part of the strategy.

Reviewer_9hCJOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper introduces hierarchical W-learning, a framework inspired by projective simulation that selects strategies and actions through successive conditional policies at each environment step. It defines strategy-prefix values and develops value-based and policy-gradient variants. Experiments on gridworld navigation, continuous navigation, and four MuJoCo tasks examine performance with designed hierarchies and feedback.

Critical strengths
  1. The strategy-prefix value formulation provides an explicit framework connecting hierarchical action selection with value-based and policy-gradient learning. This offers a useful modeling perspective for tasks with meaningful strategy structure.
  2. The gridworld experiments examine feedback magnitudes, all 16 sign patterns, and several algorithmic variants. These analyses provide useful evidence about the conditions under which the designed hierarchy and feedback help.
Critical weaknesses
  1. Theorem 4 equates convergence to an optimal policy with the value-function gradient tending to zero. However, a vanishing gradient does not establish global optimality. Please clarify the convergence result actually proved and revise the theorem and related optimality claims accordingly.
  2. The experiments introduce the hierarchy, predefined strategies, and specially designed feedback together, leaving the sources of the performance gains insufficiently separated. Although Figure 4(c) provides some relevant controls, the contributions of these factors remain unclear in the MuJoCo experiments. A controlled comparison on a representative task would help establish the contribution of the hierarchy itself. The descriptions of HA2C, HPPO, and HSAC omit key implementation details. In particular, please specify how the HPPO probability ratio and clipping objective, and the HSAC entropy term, are applied to the hierarchical policies and how each level is updated. Please provide the corresponding loss functions and update rules.
Questions for authors
  1. Clarification: Does Theorem 4 establish convergence to a stationary point or to a globally optimal policy?
  2. Additional experimental evidence: Can a representative MuJoCo comparison isolate the hierarchy’s contribution from the predefined strategies and feedback?
  3. Implementation clarification: What are the hierarchical HPPO and HSAC objectives, and how are their terms and updates defined at each level?
3 comments

1.The global optimality comes from the converge of globally defined W function in Theorem 3, because the optimal W function is the unique fix point of the Bellman optimality operator. Then we argue the induced optimal policy is global.

2.Experiments have been done on four representative MuJoCo tasks shown in appendix H. Two of them show that the structure has negative effect. One shows positive effect while another implies that the structure has no effect. The structure on MuJoCo is almost trivial, also in practice, we should split the batch into two parts to train the two strategies. Those are the reasons why the structure contribution is limited.

3.Implementions of HSAC and HPPO are shown in appendix B.4 and B.5. For HSAC, we start form the decomposition of the entropy between two levels and obtain the information propagating between them. Then we construct the corresponding soft sate value V and strategy value W functions, which enables us to update the training parameters level by level. For HPPO, we apply a clip on the strategy policy between two levels and then directly generalise the standard objective to multi-level learning.

Thank you for the rebuttal. The references to Appendices B.4 and B.5 help clarify the implementation, and the mixed MuJoCo results suggest that the benefits of the hierarchy vary by task. I remain uncertain about what convergence guarantee Theorem 4 establishes for the policy gradient updates. I maintain my rating of 2.

Sorry we misunderstood your question. The update converges to a stationary point and we have revised the statement in Theorem 4 ( page 13).

Reviewer_3s9uOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes a hierarchical reinforcement learning framework called "Hierarchical W-Learning." The framework extends the traditional Q-function into a W-function, enabling the evaluation of policies across multiple hierarchical levels. The authors introduce hierarchical temporal-difference learning—incorporating level-specific feedback—and derive a hierarchical policy gradient theorem. This framework can be integrated into existing reinforcement learning algorithms, such as DPG, A2C, PPO, and SAC. Experiments conducted in grid-world navigation, continuous-space navigation, and MuJoCo benchmarks demonstrate that this framework offers improvements in both learning efficiency and performance compared to traditional reinforcement learning methods. However, its effectiveness relies heavily on manually designed policy hierarchies and carefully tuned feedback parameters, which to some extent limits its general applicability.

Critical strengths
  1. This paper proposes a general framework that extends traditional Q-learning into "hierarchical W-learning," enabling agents to make decisions through multi-layered policies. The framework provides a systematic approach for integrating hierarchical decision-making mechanisms into various RL algorithms.

  2. The authors establish a mathematical foundation for the proposed framework. This theoretical construction renders the framework more systematic than hierarchical architectures based purely on heuristics; however, the novelty and scope of its theoretical guarantees warrant further substantiation.

  3. One of the paper’s most significant contributions to practical application is the extension of this hierarchical framework to a range of established RL methods, including REINFORCE, Actor-Critic, DPG, A2C, PPO, and SAC. This compatibility with diverse RL algorithms demonstrates the framework's flexibility in practical settings and suggests a broad potential for application.

Critical weaknesses
  1. Compared with previous HRL papers, the novelty of this work is limited and not enought for major conferences like ICLR.

The hierarchical W-function and policy gradient formulation proposed in the paper overlap significantly with existing HRL methods, such as the Options framework, MAXQ, Option-Critic, and Hierarchical Option-Critic.

Specifically, hierarchical policy decomposition is a standard approach in hierarchical decision-making. The W-learning update mechanism presented in the paper is largely analogous to applying conventional Q-learning across multiple levels. Furthermore, the hierarchical policy gradient theorem appears to follow existing policy gradient formulations.

The study fails to clearly articulate what unique capabilities W-learning actually provides, nor does it explain how it differs fundamentally from existing hierarchical Actor-Critic methods.

  1. The proposed framework requires predefined strategy hierarchies and manually specified level-specific feedback functions

For example, in the gridworld experiments, the authors manually define two strategies: moving toward the goal and avoiding obstacles. They also assign handcrafted positive or negative feedback depending on the chosen strategy and environmental outcome.

The method's effectiveness depends on prior knowledge that may not be available in general RL applicaitons, substantially limiting its generality.

  1. There are insufficient comparisons with existing hierarchical RL algorithms. In the experimental evaluation, the researchers primarily compared their hierarchical variants against corresponding non-hierarchical baseline algorithms, specifically Q-learning, DPG, A2C, PPO, and SAC.

However, the paper lacks direct empirical comparisons with established HRL algorithms, such as Option-Critic, Hierarchical Option-Critic, and MAXQ.

Such comparisons are crucial, as the core claim of this research centers on the advantages of a novel HRL framework, rather than merely the benefits of a hierarchical structure over flat policies.

  1. The paper fails to sufficiently disentangle the independent contributions of its various components, not providing sufficient ablation study.

Specifically, authors should provide comparison between hierarchical policy architectures that include "level-specific feedback" and those that do not. In addition, the comparison between W-learning and a conventional hierarchical actor-critic [1] implementation that employs the same hierarchical structure.

[1] Learning Multi-Level Hierarchies with Hindsight. ICLR 2019

Besides, authors should also provide the comparison across different hierarchical depths where the model capacity and computational budget are kept constant.

Although the authors explore different settings for the feedback parameters, these experiments do not fully isolate the benefits derived from "hierarchical representations" from the effects stemming from "reward shaping" and "architectural constraints."

  1. The evaluation domains in this paper are quite limited. The experiments focus mainly on simple and toy navigation environments and a small selection of MuJoCo control tasks which are almost the MNIST in RL research.

Further, the paper does not adequately investigate how the framework scales with increasing hierarchy depth, number of strategies, state-space complexity, or task diversity.

Questions for authors
  1. What is the fundamental difference between the proposed hierarchical W-function and the existing hierarchical value functions found in frameworks such as "Options," MAXQ, or the "Hierarchical Option-Critic"? Can the authors demonstrate that this approach offers theoretical or algorithmic advantages that cannot be achieved by traditional hierarchical actor-critic methods?

  2. Given that the performance gains rely on manually designed policy hierarchies and feedback signals tailored to specific levels, would the proposed method still outperform traditional reinforcement learning algorithms if the baselines were provided with the same domain knowledge and reward shaping?

  3. Does the introduction of additional hierarchies always improve learning performance? Or, given identical architectural constraints and reward shaping, could the same level of performance improvement be achieved using only a single-layer policy?

2 comments

We do agree there are lots of experiments to be done, but at the same time, we must argue that there are fundamental differences between W-learning and other HRL methods, including the method showed in [1], as we will explain.

1a)There is no termination function in W-learning compared with the standard HRL methods like Option framework, MaxQ and option-critic methods. Also, the optimal policy determined by the optimal W function is global. One can see more discussions in section4.

1b)Compared with the Hierarchical Actor Critic method showed in [1]. The fundamental different is that we can simultaneously study multiple goals at each level and learn the transition between them. In the HierQ update procedure, there is one Q function at each level. But for W-learning, we have different Q functions at each level and update them at the same time. This is what we called W functions. There are other W functions at higher level determining the transition between lower levels goals.

Taking the ant navigation task for example, the goal is to reach the yellow square. But pushing the ant two quickly to the goal may lead the ant to an unhealthy state. To deal with such problem. We need to introduce another goal, which is keeping the ant at a proper position so it will not crash. Moreover, we need to introduce a higher-level strategy to decide the priority between the goals at each state, going to the yellow square or keep in a healthy position. This is like the work we have done in the simpler Walker2d and Hopper task. There should be a balance between going along the x direction and keeping the action small.

2,3) We try to address the problem 2 and 3 from two aspects:

For the strategy structure: Experiments have been done on four representative MuJoCo tasks in appendix H, which shows that the structure could have different effects on different tasks. Experiments results imply that a deeper level structure may play a negative role when the batch size is fixed.

For the reward shaping: There are two ways of shaping in each one-level strategy. We can apply one of them on the standard RL training and evaluate the performance using the original reward. It is possible to get better performance but that is less stable. There is no theoretical support for us to do so, training with a set of rewards and evaluate the performance on the other set.

Thanks for author's response and revision. It is a promising direction to go, but the current version of this paper is not enough for ICLR. I keep my score

Reviewer_92khOpenReview ↗
Rating
2
Confidence
2
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes a simple and general framework for hierarchical reinforcement learning, which extends the conventional action-value Q-function to a hierarchical W-function, enabling an agent to select actions according to a hierarchy of strategies. Experimental results demonstrate that well-designed hierarchical strategies and properly chosen update parameters consistently improve reinforcement learning performance.

Critical strengths
  1. This paper provides a mathematical formulation of hierarchical W-learning, including hierarchical policy gradient theorems and convergence analysis.
  2. This paper extends the hierarchical framework to multiple reinforcement learning algorithms and demonstrates performance improvements across discrete and continuous control tasks.
Critical weaknesses
  1. The proposed framework shares substantial similarities with existing hierarchical reinforcement learning methods, and its distinct contributions are not sufficiently established.
  2. Performance improvements depend heavily on manual design including reward parameters, limiting generalizability and practical applicability.
Questions for authors

1.What are the fundamental theoretical and algorithmic differences between W-learning and existing hierarchical reinforcement learning frameworks? 2.How do the convergence guarantees extend to deep reinforcement learning algorithms with nonlinear function approximation? 3.How can effective hierarchical strategies be constructed for complex tasks without relying on extensive domain knowledge or manual design? 4.How sensitive is W-learning to the choice of hierarchical feedback functions g_k, learning rates, and discount factors across different environments? 5.Why are established hierarchical reinforcement learning methods not included as baselines, and how would W-learning compare against them under equivalent experimental settings? 6.Can the proposed framework maintain its performance advantages on more challenging tasks where suitable hierarchical structures are not immediately apparent?

1 comment

1)Compared with the Option framework, MaxQ and option-critic methods, the absence of termination function makes our update rules simple and easier to generalise to a wide range of tasks. Fundamentally, we can say that the dynamics of the previous HRL methods specified by the termination function now is encoded in the design of strategy architecture and update parameters. The optimal policy determined by W-learning is global. More discussions are shown in section4.

2)The proof for the convergence of deep Q learning is done in the work [1][2] under some assumption for simplicity. The proof for the deep W learning case goes beyond our paper but we think similar results also hold. We present some theoretical discussions and the implementation detail on HPPO and HSAC in the appendix B.5 and B.4 without proofs on their convergence.

3)For tasks with sparse reward, we can set intermediate rewards to encourage the exploration, for example the difference between current state and next state. For continuous control tasks, one can construct functions which are maximal when the state is near stable position, or with small velocity, and set them as the parameter g. It is also useful to look at the reward designed by the benchmark.

4)In the context of discrete navigation, the agent works well in a wide range of parameters. 12 out of 16 combinations of feedback parameters show advantage. That is because the structure is well designed. In general, the overall learning speed depends on the high- level learning rates.

For continuous navigation, the combination of learning rates and feedback parameters plays an important role. The agent is likely to stuck at the boundary or the obstacles so that the training is frozen. To deal with such problem, we try to decrease the critic learning rate, reduce the critic update frequence and increase the ratio of the feedback parameters so that the agent can quickly switch between two strategies when hitting the obstacle.

For MuJoCo tasks, the sensitivity of the feedback parameters varies among different algorithms on different tasks. The main reason is that we are using an one-level trivial structure. For the easier task with sparse reward like InvertedPendulum, a wide range of parameters works well. For more complicated tasks with delicate reward design, the performance becomes sensitive to the feedback.

5)As far as we know, no HRL learning method claims it has equivalent or better performance than the A2C, PPO and SAC baselines on the Mujoco tasks we studied in the paper.

6)The question is related to strategy discovery. In our framework, we provide two ways to develop the structure. One is building the structure from smaller one which has been pre-trained in easier situation, the other is cutting off legs from a bigger structure. For example, in the navigation task, one can first train the agent when there is no obstacle and add higher-level strategies when the agent is put into the environment with obstacles. Or we can directly start from a fully connected high-level structure and then cut off the legs with low transition probability but keep all actions connected.

[1] On the convergence and sample complexity analysis of deep Q networks with ϵ\epsilon greedy exploration

[2] A Theoretical analysis of deep q-learning

Comments on the paper

1 comment

Two main modifications have been made in the new version:

  • Four representative MuJoCo experiments have been conducted in appendix H, in order to illustrate the contribution from the strategy structure.

  • Implementations of the HSAC and HPPO, together with some theoretical constructions, have been added in appendix B4 and appendix B5.

Some reviewers asked a common crucial question: “What is the fundamental difference between W-learning and other hierarchical learning methods?”. We claim three differences:

  • No termination in our work, which makes it easier to apply on various tasks.

  • The optimal policy determined by W function is global.

  • It is possible to simultaneously learn multi goals at each level and the transitions between them.

These three advantages make W-learning a unique approach to hierarchical reinforcement learning.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.