- Rating
- 1
- Confidence
- 4
- Informativeness
- 1
- Presentation
- 1
- Soundness
- 1
Read review
Paper summary
This paper is on hierarchical reinforcement learning with the options framework. This topic was quite popular around 2020~2022, and the related work section shows some relevant papers in the literature, but the list can be expanded to include HRL/options framework paper that integrates the symbolic approach and the deep RL approaches.
The main issue with the current submission is that this paper doesn't formally define concepts and notations as well. I was not able to follow up and understand the paper. I will post the clarification questions below.
From what's written, the proposed approach is closely related to SCALAR (if so, this needs to be provided as background for clear understanding), doing HRL with integration of symbolic approach and deep RL with recurrent state encoder. The proposed algorithm is evaluated on a few instances in MiniHack, showing better performance than SCALAR.
Critical strengths
MiniHack is considered difficult. The positive strength is that the proposed paper shows better performance than the baseline SCALAR in the selected problem sets.
Critical weaknesses
The most critical weakness is that this paper does not clearly describe the proposed approach, algorithm, and why it worked well.
Questions for authors
1 Introduction and 2 Related Works
The topic of this paper is option critic and hierarchical reinforcement learning. Where the current submitted paper stands, compared with related works? The paragraphs in the related works show a good criteria. In terms of hierarchical control, skill discovery (option learning), symbolic integration, skill (option) selection, how the proposed approach is different from other approaches?
The paper mentions SCALAR several times (line 132, line 145, line 160, line 262), is this paper relying on the SCALAR? how much?
3 Planning Guided Option Critics
In hierarchical RL, they always have mechanisms to make a connection between the abstract, higher-level and the lower level that acts on the states with primitive actions. In this paper, how is it achieved?
"PGOC composes symbolically specified skills with a learned option critic rather than a fixed plan" Could you explain how this is done?
How is the symbolically specified skill obtained? What is the difference between a skill and an option? From what's written, they are the same, where a skill is defined as a triple of initiation, termination, and reward.
When it learns a policy, on what level is the policy learned? at the symbolic-abstraction state level? or at the state level? Does each option (or skill) need to learn its own policy to terminate the skill/option?
3.1. Symbolic Skill Specifications How is the symbolic abstraction (z_t = \phi(s_t)) learned? In Algorithm 1, Skills are alreday the input.
Eq. (1) seems like it is a set of options that satisfy the initiation set and additional option predicate N_o(z_t, u_t). u_t is undefined at this point. The notation seems to work at the abstraction level, seeing that initiation and termination is defined over z_t and z_{t+1}. How is the reward function defined?
In Algorithm 1, there are several symbols not defined. \mu seems like an option selection policy, how do you initialize it? what is q_{t, j}? how r^{o_t}_{t} is defined?
3.2. Plan Progress Requirement
What is requirement j? What is B_j? What is one unit in "when each successful execution produces one unit"? how do we know q_{t,j}?
3.3 Advantage estimation How h_t and z_t are different? How do we get h_t?
Does the \mu and V^{\mu} define policy and value at the abstract level through the recurrent states? How is such a state encoder defined?
What is the difference between V^{\mu} and V_o ? When the same skill library set was used as shown in Appendix I, does it imply that the skills were trained before applying Algorithm 1?
4 Experiment Can you describe the benchmark with proper citations? How the problem sets were chosen, and what is the rationale for working on those problems?
- Use of AI Can you speciify what are the LLM-generated research claims and how they were checked?
5 comments
Response to Reviewer bLVt (Part 2)
From what's written, the proposed approach is closely related to SCALAR (if so, this needs to be provided as background for clear understanding), doing HRL with integration of symbolic approach and deep RL with recurrent state encoder.
The paper mentions SCALAR several times (line 132, line 145, line 160, line 262), is this paper relying on the SCALAR? how much?
PGOC relies on SCALAR for its inputs, not for its learning method. It adopts SCALAR's format for skill specifications (Section 3.1). On Craftax it uses SCALAR's skill specifications themselves, so the comparison between the two holds the specifications fixed (Appendix I). SCALAR is also our main baseline.
How the skills are learned and composed differs. SCALAR executes skills in a fixed planned order and passes credit across skill boundaries by bootstrapping from the next skill's value. PGOC learns the order with an observation-conditioned controller, bootstraps from the controller's value only when a skill succeeds, and masks options whose preconditions do not hold (Section 2, lines 144–151). PGOC also keeps the specifications fixed and starts every episode fresh. It does not use SCALAR's refinement of specifications from trajectories or its resets to states where preconditions first hold.
We agree that the paper should be readable without SCALAR's. We have added Appendix L, which summarizes SCALAR's specification format and walks through one Craftax skill as an example, and Section 3.1 now points to it.
In hierarchical RL, they always have mechanisms to make a connection between the abstract, higher-level and the lower level that acts on the states with primitive actions. In this paper, how is it achieved?
"PGOC composes symbolically specified skills with a learned option critic rather than a fixed plan" Could you explain how this is done?
The higher level is the controller and the lower level is the set of skill policies . They are connected in two directions.
Control passes down through option selection. At a selection step, samples an option from the admissible set , the options whose preconditions hold and whose requirements are not yet met (Eq. 1, line 203). That option's policy then takes primitive actions until its completion predicate holds, and control returns to for the next selection (Algorithm 1). An execution that does not succeed ends with the episode.
Credit passes up through the skill's reward. When a skill succeeds, it receives the controller reward earned during its execution and the controller's value for what follows (Section 3.3, Eq. 6). Each skill is thus trained for what its success enables at the higher level.
"Rather than a fixed plan" refers to SCALAR, which executes the skills in an order fixed before any interaction. PGOC replaces that order with , which chooses the next skill from the agent's current observations. Section 4.4 compares the two. A fixed order does better on Craftax, where the prerequisites already determine the order (69.38% vs 62.80% Mines success). The learned controller does better on randomized TreasureDash, where the best order depends on the layout (22.21 vs 10.88).
How is the symbolically specified skill obtained? What is the difference between a skill and an option? From what's written, they are the same, where a skill is defined as a triple of initiation, termination, and reward.
On Craftax and Quest-Hard, the specifications are SCALAR's. A language model proposed them, and SCALAR refined them from trajectories. TreasureDash and ZombieHorde do not appear in SCALAR. For these we build each library from the two option rewards SOL defines for the task: gold and the staircase signal for TreasureDash, and the task score and health for ZombieHorde. Each skill is trained on one of these rewards, and we add the initiation and completion predicates that sequence them (Appendix I). PGOC holds all of these fixed. Appendix I lists them, and the new Appendix L walks through one. They could equally be written by hand, as the new Problem setting paragraph notes.
You are right that a skill and an option are the same object. As you describe, the specification consists of an initiation predicate , a completion predicate and a reward . Adding the learned policy gives an option in the standard sense of Sutton et al. (1999), with termination given by . Section 3.1 states that "we refer to skills as options when they act as units of selection" (lines 186–188).
Response to Reviewer bLVt (Part 3)
When it learns a policy, on what level is the policy learned? at the symbolic-abstraction state level? or at the state level? Does each option (or skill) need to learn its own policy to terminate the skill/option?
All policies are learned at the state level. The controller and each skill policy choose actions from , the representation the policy network computes from the agent's observations (Section 3.1, lines 176–181; Section 3.3, lines 244–247). On Craftax the observation is the environment's symbolic observation: the map around the agent together with its inventory and vital statistics (Appendix F.2). On MiniHack it is a glyph crop centered on the agent, the status line, the most recent message and the previous action (Appendix I). The symbolic abstraction is used to evaluate the specifications: which options are admissible (Eq. 1, line 203), when a skill has succeeded, and what reward it receives. The controller also receives the admissibility mask and the progress records as inputs (lines 244–247). There is no separate policy over .
No option learns its own termination. A skill ends when its completion predicate holds, and control then returns to . That predicate is part of the given specification, and an execution that never succeeds ends with the episode (Section 3.1, lines 190–194). Each therefore learns only how to reach its completion condition. Thank you for this question; we have added the second of these rules to Section 3.1, where it was previously left implicit.
3.1. Symbolic Skill Specifications How is the symbolic abstraction (z_t = \phi(s_t)) learned? In Algorithm 1, Skills are alreday the input.
The abstraction is not learned. It is a fixed function of the environment state, supplied together with the skill specifications that are evaluated on it. On Craftax, includes inventory counts, the current dungeon level and task counters such as dungeon-floor defeats (Appendix L). The Problem setting paragraph of Section 3 now states this as an assumption: the specifications are given, and they are evaluated on a symbolic abstraction of the state (lines 167–172). This is also why Algorithm 1 takes the skills as input. Its input is the specification of each skill, the triple . What PGOC learns is listed on the algorithm's initialization line: the controller with its value , and each skill's policy with its value .
Eq. (1) seems like it is a set of options that satisfy the initiation set and additional option predicate N_o(z_t, u_t). u_t is undefined at this point. The notation seems to work at the abstraction level, seeing that initiation and termination is defined over z_t and z_{t+1}. How is the reward function defined?
Your reading of Eq. 1 is correct. contains the options whose initiation predicate holds and whose continued-need predicate holds (line 203). The record is introduced just before it, in Section 3.1 (lines 180–182): it records the progress already credited toward each requirement during the episode. The formulation allows to read ; on Craftax, the continued-need predicates instead read counters kept in , such as the episode's count of floor defeats (Appendix G), so the option that fights on the first dungeon floor stays needed until that count reaches eight.
You are also right that the specifications work at the level of the abstraction. Initiation, completion and the skill rewards are all functions of and . This is intended. The specifications describe the task in terms of , and the policies act from , as described in our previous comment.
Each skill's reward is part of its specification. It combines a sparse reward for the skill's objective, such as the increase in an inventory count, with SCALAR's local shaping terms. Those terms are a vital term for changes in health, food, drink and energy, the lava and death terms, and, for some skills, a reward for reducing the distance to a target (Appendix I.1, "Local objectives"). Appendix L works through the reward of the wood-sword skill.
Response to Reviewer bLVt (Part 1)
Thank you for the detailed questions, and for noting PGOC's improvement over SCALAR on difficult tasks. Your main concern is that the method and its notation are not formally defined, and that the paper relies on SCALAR without describing it. Your questions helped us make the presentation self-contained. The revised PDF adds a Problem setting paragraph (lines 167–172), definitions of ht and zt where they are first used (lines 176–179), definitions of rto and of when an execution ends (lines 186–194), Appendix L, which describes SCALAR's specification format with a worked example, and Appendix M, which describes the benchmarks with citations and explains why they were chosen. The answers that follow cite the revised line numbers.
The main evaluation is on Craftax rather than MiniHack, and the task there is to reach the Gnomish Mines. Reaching them requires a long prerequisite chain, including eight dungeon-floor defeats to unlock the ladder. With the same skill specifications as SCALAR, PGOC reaches the Mines in 64.53% of evaluation episodes with a GRU backbone and 62.80% with the Hybrid backbone. SCALAR reaches them in 9.35% and 7.66%, and PPO in 0.00% and 4.32%. That is roughly seven times SCALAR's success rate, and the advantage holds in every matched recurrent and Hybrid seed (Section 4.1, lines 364–366). These results use five training seeds per configuration at 1B interactions, with 10,240 fresh evaluation episodes per seed (Section 4, lines 344–345). We also evaluate on Craftax-Classic and three MiniHack tasks. On randomized TreasureDash, PGOC scores 22.21 against 10.08 for SCALAR and 14.36 for SOL (Table 1).
Thank you. Section 2 already covers several such methods: PEORL, SDRL, Illanes et al. (2020) and reward machines (lines 122–134). In response to this review and the others, we have added h-DQN, ALisp and DYLAN to the revised Related Work.
Section 2 (lines 98–151) organizes related work along the lines you list.
For hierarchical control, PGOC's controller selects an option, which then acts until its success condition holds and control returns to the controller; an execution that does not succeed ends with the episode, when the agent dies. The options' initiation conditions, success conditions and rewards are all supplied. Option-Critic, by contrast, learns its terminations from the task reward. FeUdal Networks and HIRO set goals in a learned embedding or in state space.
PGOC does no skill discovery. The specifications are given, and only the skill policies are learned. Skill-discovery methods such as DIAYN and DADS learn the repertoire itself.
For symbolic integration, predicates on the symbolic state define which options may be selected and how the controller is rewarded for progress. No planner fixes the order of skills. Planner-guided methods such as PEORL and SDRL execute options in a planner-determined order, and reward machines fix the order in an automaton.
For selection, PGOC's controller is learned from observations, jointly with the skills. SCALAR executes a fixed planned order, SayCan scores skills by language relevance, and MaestroMotif selects with a frozen language model.
What none of these share is PGOC's credit assignment. A skill that succeeds is credited with the controller's value for what follows, and a skill that fails is not (Section 3.3).