- Rating
- 1
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
This paper present a novel hierarchical reinforcement learning algorithm inspired by some concept from projective simulation. The authors propose a framework that generalize flat action-value function to new hierarchical W-function, letting the agent behave accordingly to an hierarchical strategy . The authors present an off-policy learning algorithm derived from Q learning and the policy gradient, and actor-critic, case by deriving the corresponding policy gradient theorem. Finally, the authors propose a deterministic PG extension of the algorithm and provide experimental results on continuous and discrete navigation tasks.
Critical strengths
- The method presented can theoretically be put on top on various existing algorithms. The authors show this strength by combining their framework with REINFORCE, DPG, A2C, PPO and SAC.
Critical weaknesses
- The presentation of the approach is unclear and confusing in several parts. The authors adopt concepts from other literature without properly explaining them. For example, the introduction presents the “clip” as a core part of the framework, but the concept disappears in the following sections.
- The setting relies strongly on prior knowledge of the task, which is quite a restrictive assumption.
- The Authors state that "[...] incorporating an appropriate hierarchical strategy can consistently improve learning performance [...]", which is not shown in any part of the work. Results from Figure 6 shows the same performance regardless of using the hierarchical approach, with the shaded areas (which I assumed to be confidence intervals) always overlaps. Figure 3 and 4 are consistent with what I said as the W-learning approach is statistically equivalent for every case. The only case where the hierarchical approach is slightly better is in Fig. 3a, but at the cost of having a tuned hierarchical structure defined a priori.
- The theoretical analysis is mostly known result presented in a very confuse way and none of the assumptions they rely on are stated or cited. Thm. 3 proves convergence to a that was not characterized by the Authors.
Questions for authors
- Can the Authors share more details on the implementation of the hierarchical version of the baselines (PPO, A2C, SAC)?
- Can the Authors share more details about the definition of ? It is presented as a core parameter, but it is still unclear to me how it is determined.
- Regarding the continuous experimental section, what are the hyper parameters and g function of all methods? Only the inverted pendulum ones are shown.
- How are the hierarchy and the strategy defined in practice? No details have been shared on this part.
- Generally, can the Authors define clearly the framework? A lot of parts remain unclear to me. What is a strategy and how should be defined? How does the policy function work given a set of parameters?
3 comments
Thank you for your responses and the revision on the manuscript. However, I am still not convinced by the approach.
As the plot of Fig. 3, 4 (in 6 only the HPPO show some improvement w.r.t. its flat version, the other methods are identical) might suggest some type of improvement, this is mostly due to a specific optimization based on prior knowledge (g parameter) and not on the hierarchical structure. In Fig. 24 this phenomena can be seen clearly as all the results with perform equally or worse w.r.t. the flat version (HA2C is still not defined as an algorithm). This makes me raise some concerns about the usefulness of the approach.
Moreover, as stated before the theory presented is an application of standard results and not a novel results brought by the Authors. I also have some concerns on the Proof of Thm. 3 as the operator H in Appendix A uses instead of the reward. Its fixed point is therefore the value of the current policy under , not the optimal value of the task. Can you elaborate on this to help me understand better your proof?
Finally, can you elaborate on this statement done in section 4 "It enables us to find stronger policy compared to the globally optimal policy by constructing more complicated clip structure."? How can you find a policy stronger than the optimal one?
Overall, because of the rigidity of the approach, which forces the practitioner to hand-design several parts of the algorithm, in addition to what was already said, I keep my score.
-
Good question. One fact we should note that different strategies may lead to the same action, therefore the strategy is not unique given the optimal policy. The parameters introduced here are used to uniquely determine the strategy. In practice, we set the first level or the last level in order to get the optimal value of the task. But that rule is not rigid. Exploring how to determine the value of so that the agent could perform better is one of the central goal of our work. The proof holds for all bounded , because it will be cancelled when we calculate .
-
Suppose we have two strategies. I is following the standard RL policy and the other II is to do something else following other structure. During the training, if the agent finds it will get more reward following I, then it will choose the standard policy ohterwise the agnet will choose strategy II. So we can see, the learned policy is a least as strong as the standard one because that is part of the strategy.
Clarification:
“Clip’’ is the terminology used to discuss the contexts related to the projective simulation model while in the main part of the paper it is equivalent to the word “node” in the graph.
For the plot issue, sometimes we put many curves in one figure. It is better to compare green and gray, red and blue curves in figure 3a 3b, and compare gray curves with other three in fig. 3c. Also, for figure4, it is better to compare green and gray, red and blue curves in figure 4a, and compare the gray curve with other two in fig4.b, fig4,c. In figure 6a and figure 6b, it is better to compare the blue/red curves with the green/gray curves. For MuJoCo, we conducted four more representative experiments in appendix H to highlight the differences. We did average on all samples rather than some of the best trials, so the overlaps between confidence intervals sometimes are big.
The convergence proof is the standard textbook approach as we have cited in the paper, applied to our new defined W function with additional hierarchical structure.
For questions:
Implementations of the HSAC and HPPO, together with some theoretical constructions, have been added in appendix B4 and appendix B5.
The defintion of the paramter g deponds on the task. For tasks with sparse reward, we can set intermediate rewards to encourage the exploration, for example the difference between current state and next state. For continuous control tasks, one can construct functions which are maximal when the state is near stable position, or with small velocity, and set them as the parameter g. It is also useful to look at the reward designed by the benchmark.
There are three methods over four MuJoCo experiments so we have 12 sets of parameters for g. All of them take the form
g=R×C×(A+Bx) if we want a bigger x, for example velocity in x direction for Walker2d or Hopper, or g=R×A+BxC if one requires a small x, for exmaple, small actions, angles, angular velocities.
R is the reward and A, B,C are constants to be determined.