acceptodds
Under review as a conference paper at ICLR 2027

Planning Guided Option Critic

Abstract

A symbolic skill specification describes when a skill may run, what it must achieve, and how it is rewarded, but says nothing about how to learn the skills or how to choose among them. We introduce Planning Guided Option Critic (PGOC), which learns the skills and an option-critic controller jointly from the specifications alone, deriving masks and plan-level rewards directly from them. Preconditions and continued-need predicates restrict which options may be selected; plan-progress rewards credit the controller for satisfying remaining requirements; and each completed skill bootstraps from the controller's continuation value, crediting a skill for the opportunities it creates rather than its position in a fixed plan. Across five training seeds on Craftax, PGOC with recurrent backbones reaches the Gnomish Mines in 62.8–64.5% of frozen evaluation episodes, versus 7.7–9.3% for SCALAR and at most 4.3% for PPO. Ablations that retain the critic while varying the selection rule decompose this gain by task structure. On Craftax, where a fixed plan already suffices, prescribing that plan raises Mines success further to 69.4%: the credit assignment, not the learned ordering, drives the fixed-plan task. On randomized TreasureDash, where the optimal plan depends on the encountered state, the learned controller is decisive, scoring 22.2 against 10.9 for the prescribed order and 13.3 for uniform selection. The same option critic thus supplies a single credit assignment for both kinds of task, while the value of learning the selection rule itself depends on the task.

Primary area:
neurosymbolic & hybrid AI systems (physics-informed, logic & formal reasoning, etc.)
Keywords:
Hierarchical reinforcement learning, neuro-symbolic planning, temporal abstraction, option learning, long-horizon decision making, credit assignment

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_bLVtOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

This paper is on hierarchical reinforcement learning with the options framework. This topic was quite popular around 2020~2022, and the related work section shows some relevant papers in the literature, but the list can be expanded to include HRL/options framework paper that integrates the symbolic approach and the deep RL approaches.

The main issue with the current submission is that this paper doesn't formally define concepts and notations as well. I was not able to follow up and understand the paper. I will post the clarification questions below.

From what's written, the proposed approach is closely related to SCALAR (if so, this needs to be provided as background for clear understanding), doing HRL with integration of symbolic approach and deep RL with recurrent state encoder. The proposed algorithm is evaluated on a few instances in MiniHack, showing better performance than SCALAR.

Critical strengths

MiniHack is considered difficult. The positive strength is that the proposed paper shows better performance than the baseline SCALAR in the selected problem sets.

Critical weaknesses

The most critical weakness is that this paper does not clearly describe the proposed approach, algorithm, and why it worked well.

Questions for authors

1 Introduction and 2 Related Works

The topic of this paper is option critic and hierarchical reinforcement learning. Where the current submitted paper stands, compared with related works? The paragraphs in the related works show a good criteria. In terms of hierarchical control, skill discovery (option learning), symbolic integration, skill (option) selection, how the proposed approach is different from other approaches?

The paper mentions SCALAR several times (line 132, line 145, line 160, line 262), is this paper relying on the SCALAR? how much?

3 Planning Guided Option Critics

In hierarchical RL, they always have mechanisms to make a connection between the abstract, higher-level and the lower level that acts on the states with primitive actions. In this paper, how is it achieved?

"PGOC composes symbolically specified skills with a learned option critic rather than a fixed plan" Could you explain how this is done?

How is the symbolically specified skill obtained? What is the difference between a skill and an option? From what's written, they are the same, where a skill is defined as a triple of initiation, termination, and reward.

When it learns a policy, on what level is the policy learned? at the symbolic-abstraction state level? or at the state level? Does each option (or skill) need to learn its own policy to terminate the skill/option?

3.1. Symbolic Skill Specifications How is the symbolic abstraction (z_t = \phi(s_t)) learned? In Algorithm 1, Skills are alreday the input.

Eq. (1) seems like it is a set of options that satisfy the initiation set and additional option predicate N_o(z_t, u_t). u_t is undefined at this point. The notation seems to work at the abstraction level, seeing that initiation and termination is defined over z_t and z_{t+1}. How is the reward function defined?

In Algorithm 1, there are several symbols not defined. \mu seems like an option selection policy, how do you initialize it? what is q_{t, j}? how r^{o_t}_{t} is defined?

3.2. Plan Progress Requirement

What is requirement j? What is B_j? What is one unit in "when each successful execution produces one unit"? how do we know q_{t,j}?

3.3 Advantage estimation How h_t and z_t are different? How do we get h_t?

Does the \mu and V^{\mu} define policy and value at the abstract level through the recurrent states? How is such a state encoder defined?

What is the difference between V^{\mu} and V_o ? When the same skill library set was used as shown in Appendix I, does it imply that the skills were trained before applying Algorithm 1?

4 Experiment Can you describe the benchmark with proper citations? How the problem sets were chosen, and what is the rationale for working on those problems?

  1. Use of AI Can you speciify what are the LLM-generated research claims and how they were checked?
5 comments
AuthorsOpenReview ↗

Response to Reviewer bLVt (Part 1)

Thank you for the detailed questions, and for noting PGOC's improvement over SCALAR on difficult tasks. Your main concern is that the method and its notation are not formally defined, and that the paper relies on SCALAR without describing it. Your questions helped us make the presentation self-contained. The revised PDF adds a Problem setting paragraph (lines 167–172), definitions of hth_t and ztz_t where they are first used (lines 176–179), definitions of rtor^o_t and of when an execution ends (lines 186–194), Appendix L, which describes SCALAR's specification format with a worked example, and Appendix M, which describes the benchmarks with citations and explains why they were chosen. The answers that follow cite the revised line numbers.

The proposed algorithm is evaluated on a few instances in MiniHack, showing better performance than SCALAR.

The main evaluation is on Craftax rather than MiniHack, and the task there is to reach the Gnomish Mines. Reaching them requires a long prerequisite chain, including eight dungeon-floor defeats to unlock the ladder. With the same skill specifications as SCALAR, PGOC reaches the Mines in 64.53% of evaluation episodes with a GRU backbone and 62.80% with the Hybrid backbone. SCALAR reaches them in 9.35% and 7.66%, and PPO in 0.00% and 4.32%. That is roughly seven times SCALAR's success rate, and the advantage holds in every matched recurrent and Hybrid seed (Section 4.1, lines 364–366). These results use five training seeds per configuration at 1B interactions, with 10,240 fresh evaluation episodes per seed (Section 4, lines 344–345). We also evaluate on Craftax-Classic and three MiniHack tasks. On randomized TreasureDash, PGOC scores 22.21 against 10.08 for SCALAR and 14.36 for SOL (Table 1).

This topic was quite popular around 2020~2022, and the related work section shows some relevant papers in the literature, but the list can be expanded to include HRL/options framework paper that integrates the symbolic approach and the deep RL approaches.

Thank you. Section 2 already covers several such methods: PEORL, SDRL, Illanes et al. (2020) and reward machines (lines 122–134). In response to this review and the others, we have added h-DQN, ALisp and DYLAN to the revised Related Work.

Where the current submitted paper stands, compared with related works? The paragraphs in the related works show a good criteria. In terms of hierarchical control, skill discovery (option learning), symbolic integration, skill (option) selection, how the proposed approach is different from other approaches?

Section 2 (lines 98–151) organizes related work along the lines you list.

For hierarchical control, PGOC's controller selects an option, which then acts until its success condition holds and control returns to the controller; an execution that does not succeed ends with the episode, when the agent dies. The options' initiation conditions, success conditions and rewards are all supplied. Option-Critic, by contrast, learns its terminations from the task reward. FeUdal Networks and HIRO set goals in a learned embedding or in state space.

PGOC does no skill discovery. The specifications are given, and only the skill policies are learned. Skill-discovery methods such as DIAYN and DADS learn the repertoire itself.

For symbolic integration, predicates on the symbolic state define which options may be selected and how the controller is rewarded for progress. No planner fixes the order of skills. Planner-guided methods such as PEORL and SDRL execute options in a planner-determined order, and reward machines fix the order in an automaton.

For selection, PGOC's controller is learned from observations, jointly with the skills. SCALAR executes a fixed planned order, SayCan scores skills by language relevance, and MaestroMotif selects with a frozen language model.

What none of these share is PGOC's credit assignment. A skill that succeeds is credited with the controller's value for what follows, and a skill that fails is not (Section 3.3).

AuthorsOpenReview ↗

Response to Reviewer bLVt (Part 2)

From what's written, the proposed approach is closely related to SCALAR (if so, this needs to be provided as background for clear understanding), doing HRL with integration of symbolic approach and deep RL with recurrent state encoder.

The paper mentions SCALAR several times (line 132, line 145, line 160, line 262), is this paper relying on the SCALAR? how much?

PGOC relies on SCALAR for its inputs, not for its learning method. It adopts SCALAR's format for skill specifications (Section 3.1). On Craftax it uses SCALAR's skill specifications themselves, so the comparison between the two holds the specifications fixed (Appendix I). SCALAR is also our main baseline.

How the skills are learned and composed differs. SCALAR executes skills in a fixed planned order and passes credit across skill boundaries by bootstrapping from the next skill's value. PGOC learns the order with an observation-conditioned controller, bootstraps from the controller's value only when a skill succeeds, and masks options whose preconditions do not hold (Section 2, lines 144–151). PGOC also keeps the specifications fixed and starts every episode fresh. It does not use SCALAR's refinement of specifications from trajectories or its resets to states where preconditions first hold.

We agree that the paper should be readable without SCALAR's. We have added Appendix L, which summarizes SCALAR's specification format and walks through one Craftax skill as an example, and Section 3.1 now points to it.

In hierarchical RL, they always have mechanisms to make a connection between the abstract, higher-level and the lower level that acts on the states with primitive actions. In this paper, how is it achieved?

"PGOC composes symbolically specified skills with a learned option critic rather than a fixed plan" Could you explain how this is done?

The higher level is the controller μ\mu and the lower level is the set of skill policies πo\pi_o. They are connected in two directions.

Control passes down through option selection. At a selection step, μ\mu samples an option from the admissible set Ot\mathcal O_t, the options whose preconditions hold and whose requirements are not yet met (Eq. 1, line 203). That option's policy then takes primitive actions until its completion predicate holds, and control returns to μ\mu for the next selection (Algorithm 1). An execution that does not succeed ends with the episode.

Credit passes up through the skill's reward. When a skill succeeds, it receives the controller reward earned during its execution and the controller's value for what follows (Section 3.3, Eq. 6). Each skill is thus trained for what its success enables at the higher level.

"Rather than a fixed plan" refers to SCALAR, which executes the skills in an order fixed before any interaction. PGOC replaces that order with μ\mu, which chooses the next skill from the agent's current observations. Section 4.4 compares the two. A fixed order does better on Craftax, where the prerequisites already determine the order (69.38% vs 62.80% Mines success). The learned controller does better on randomized TreasureDash, where the best order depends on the layout (22.21 vs 10.88).

How is the symbolically specified skill obtained? What is the difference between a skill and an option? From what's written, they are the same, where a skill is defined as a triple of initiation, termination, and reward.

On Craftax and Quest-Hard, the specifications are SCALAR's. A language model proposed them, and SCALAR refined them from trajectories. TreasureDash and ZombieHorde do not appear in SCALAR. For these we build each library from the two option rewards SOL defines for the task: gold and the staircase signal for TreasureDash, and the task score and health for ZombieHorde. Each skill is trained on one of these rewards, and we add the initiation and completion predicates that sequence them (Appendix I). PGOC holds all of these fixed. Appendix I lists them, and the new Appendix L walks through one. They could equally be written by hand, as the new Problem setting paragraph notes.

You are right that a skill and an option are the same object. As you describe, the specification consists of an initiation predicate IoI_o, a completion predicate τo\tau_o and a reward ror_o. Adding the learned policy πo\pi_o gives an option in the standard sense of Sutton et al. (1999), with termination given by τo\tau_o. Section 3.1 states that "we refer to skills as options when they act as units of selection" (lines 186–188).

AuthorsOpenReview ↗

Response to Reviewer bLVt (Part 3)

When it learns a policy, on what level is the policy learned? at the symbolic-abstraction state level? or at the state level? Does each option (or skill) need to learn its own policy to terminate the skill/option?

All policies are learned at the state level. The controller μ\mu and each skill policy πo\pi_o choose actions from hth_t, the representation the policy network computes from the agent's observations (Section 3.1, lines 176–181; Section 3.3, lines 244–247). On Craftax the observation is the environment's symbolic observation: the map around the agent together with its inventory and vital statistics (Appendix F.2). On MiniHack it is a glyph crop centered on the agent, the status line, the most recent message and the previous action (Appendix I). The symbolic abstraction ztz_t is used to evaluate the specifications: which options are admissible (Eq. 1, line 203), when a skill has succeeded, and what reward it receives. The controller also receives the admissibility mask and the progress records utu_t as inputs (lines 244–247). There is no separate policy over ztz_t.

No option learns its own termination. A skill ends when its completion predicate τo\tau_o holds, and control then returns to μ\mu. That predicate is part of the given specification, and an execution that never succeeds ends with the episode (Section 3.1, lines 190–194). Each πo\pi_o therefore learns only how to reach its completion condition. Thank you for this question; we have added the second of these rules to Section 3.1, where it was previously left implicit.

3.1. Symbolic Skill Specifications How is the symbolic abstraction (z_t = \phi(s_t)) learned? In Algorithm 1, Skills are alreday the input.

The abstraction ϕ\phi is not learned. It is a fixed function of the environment state, supplied together with the skill specifications that are evaluated on it. On Craftax, ztz_t includes inventory counts, the current dungeon level and task counters such as dungeon-floor defeats (Appendix L). The Problem setting paragraph of Section 3 now states this as an assumption: the specifications are given, and they are evaluated on a symbolic abstraction of the state (lines 167–172). This is also why Algorithm 1 takes the skills as input. Its input is the specification of each skill, the triple (Io,τo,ro)(I_o, \tau_o, r_o). What PGOC learns is listed on the algorithm's initialization line: the controller μ\mu with its value VμV^\mu, and each skill's policy πo\pi_o with its value VoV_o.

Eq. (1) seems like it is a set of options that satisfy the initiation set and additional option predicate N_o(z_t, u_t). u_t is undefined at this point. The notation seems to work at the abstraction level, seeing that initiation and termination is defined over z_t and z_{t+1}. How is the reward function defined?

Your reading of Eq. 1 is correct. Ot\mathcal O_t contains the options whose initiation predicate IoI_o holds and whose continued-need predicate NoN_o holds (line 203). The record utu_t is introduced just before it, in Section 3.1 (lines 180–182): it records the progress already credited toward each requirement during the episode. The formulation allows NoN_o to read utu_t; on Craftax, the continued-need predicates instead read counters kept in ztz_t, such as the episode's count of floor defeats (Appendix G), so the option that fights on the first dungeon floor stays needed until that count reaches eight.

You are also right that the specifications work at the level of the abstraction. Initiation, completion and the skill rewards are all functions of ztz_t and zt+1z_{t+1}. This is intended. The specifications describe the task in terms of ztz_t, and the policies act from hth_t, as described in our previous comment.

Each skill's reward ror_o is part of its specification. It combines a sparse reward for the skill's objective, such as the increase in an inventory count, with SCALAR's local shaping terms. Those terms are a vital term for changes in health, food, drink and energy, the lava and death terms, and, for some skills, a reward for reducing the distance to a target (Appendix I.1, "Local objectives"). Appendix L works through the reward of the wood-sword skill.

Reviewer_JzE9OpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

The idea of the paper using a planner to guide the agent's learning is interesting, but the paper lacks discussion of closely related work, which makes it hard to assess the paper's contribution. Also, the experimental section is relatively small, with limited baseline comparisons. Details see weakness.

Critical strengths
  • The investigated idea is interesting, and the results are promising
Critical weaknesses
  • My first concern is the missing discussion of closely related work, therefore, the novelty claims are insufficiently contextualized, which weakens the paper's contribution. PGOC proposes integrating symbolic skill specifications with learned hierarchical control and plan-guided reward signals. However, the novelty of this overall framework is not sufficiently established relative to closely related approaches. For example, DYLAN [1] already investigated the integration of symbolic planning and deep RL, including adaptive plan-based reward shaping and compositional execution of reusable policy primitives. The absence of a discussion of DYLAN and related work makes it difficult to assess the novelty of the proposed method. I would highly suggest that the authors better position PGOC in a broader discussion within related work.

  • The paper emphasizes the joint learning of the high-level option-selection policy (controller, not the planner) and low-level skill policies as a central contribution. However such simultaneous optimization of high- and low-level policies is established in hierarchical RL, including Option-Critic and SAC-X [2]. Thus, I would suggest the authors explain more explicitly the contribution and how it is distinct from previous works. Another question is that jointly learning the controller and skill may introduce a potential chicken-egg problem: the controller relies on sufficient skills to discover successful task sequences (the plan is provided externally) and estimate their continuation values, while the skills depend on the controller to select them and provide meaningful training. This leads to an issue: if an early skill fails to reach a required poststate, subsequent options may never become admissible or receive sufficient training. Consequently, the controller may struggle to learn meaningful selection policies for downstream options, while those options cannot improve without being executed. Such dependencies could lead to sparse learning signals and poor sample efficiency as the planning horizon and number of options increase, which constrains PGOC's capability. Thus, I would like to ask the authors how the option is designed? and what is the limit of PGOC in terms of scalability? Moreover, does the learned option's behavior align with its semantic meaning?

  • The assumptions and the problem definitions of PGOC are not clear: what assumptions does PGOC make, and what kind of problem does PGOC aim to address? Inadequate positioning leads to insufficient baseline comparison and ablation: for example, If we treat PGOC as an exploration paper, we are missing baseline exploration methods such as NovelD [3] and DEIR[4]. If we treat PGOC as a reward-shaping paper, we are missing reward machine [5] and Dylan [1] as comparisons. Then, for option reuse, we lack compositional generation methods such as DiffStitch [6] and TMD [7]. And how about comparison with hierarchical methods such as hDQN [8], option critic, and feudal networks, and in general, goal-conditioned RL?

  • Furthermore, how is the reward designed to obtain the desired high-level policy and low-level option? and how brittle is the reward design of the pipeline? Will PGOC learn shortcuts or fail to learn if multiple paths exist?

[1] Zihan Ye et al, Learning from Less: Guiding Deep Reinforcement Learning with Differentiable Symbolic Planning, PRL@ RLC 2025 , url= https://openreview.net/forum?id=Cm5CN0eLu6

[2] Riedmiller M, Hafner R, Lampe T, et al. Learning by playing solving sparse reward tasks from scratch[C]//International conference on machine learning. PMLR, 2018: 4344-4353

[3] Zhang T, Xu H, Wang X, et al. Noveld: A simple yet effective exploration criterion[J]. Advances in Neural Information Processing Systems, 2021, 34: 25217-25230.

[4] DEIR: efficient and robust exploration through discriminative-model-based episodic intrinsic rewards

[5] Icarte R T, Klassen T Q, Valenzano R, et al. Reward machines: Exploiting reward function structure in reinforcement learning[J]. Journal of Artificial Intelligence Research, 2022, 73: 173-208.

[6] Guanghe Li, Yixiang Shan, Zhengbang Zhu, Ting Long, Weinan Zhang Proceedings of the 41st International Conference on Machine Learning, PMLR 235:28597-28609, 2024.

[7] Myers V, Zheng B, Eysenbach B, et al. Offline goal-conditioned reinforcement learning with quasimetric representations[J]. Advances in Neural Information Processing Systems

[8] Kulkarni T D, Narasimhan K, Saeedi A, et al. Hierarchical deep reinforcement learning: Integrating temporal abstraction and intrinsic motivation[J]. Advances in neural information processing systems, 2016, 29.

Questions for authors

See weakness. I am happy to discuss with the authors to adjust my recommendation accordingly based on the rebuttal. I highly suggest the authors to better position the work and distinguish the work with prior related work.

4 comments
AuthorsOpenReview ↗

Response to Reviewer JzE9 (Part 1)

Thank you for the constructive review, and for finding the idea interesting and the results promising. Your main concerns are how PGOC is positioned relative to related work, in particular DYLAN, Option-Critic and SAC-X; what PGOC assumes and which problem it addresses; and which baselines it should be compared with. Following your review, the revised PDF adds a Problem setting paragraph stating PGOC's assumptions (lines 167–172), adds DYLAN (lines 124–125) and h-DQN (lines 105–106) to Related Work together with a comparison of how Option-Critic, SAC-X, SOL and PGOC credit their low-level policies (lines 115–120), and adds the case you raise of a skill that never succeeds to the Limitations paragraph (line 507).

My first concern is the missing discussion of closely related work, therefore, the novelty claims are insufficiently contextualized, which weakens the paper's contribution. PGOC proposes integrating symbolic skill specifications with learned hierarchical control and plan-guided reward signals. However, the novelty of this overall framework is not sufficiently established relative to closely related approaches. For example, DYLAN [1] already investigated the integration of symbolic planning and deep RL, including adaptive plan-based reward shaping and compositional execution of reusable policy primitives. The absence of a discussion of DYLAN and related work makes it difficult to assess the novelty of the proposed method. I would highly suggest that the authors better position PGOC in a broader discussion within related work.

Thank you for the pointer to DYLAN; it is a useful point of contrast, and we have added it to the revised Related Work. DYLAN trains a single flat PPO or A2C policy on the environment reward plus a planner-derived bonus for reaching plan subgoals in order. In its composition mode, the planner chooses among primitives through rule weights fit by a supervised loss. It does not train separate skills with their own rewards, and it does not learn a controller over them by RL. In PGOC, each skill is trained on its own reward, an RL controller chooses among admissible skills, and the question is how a skill should be credited when control returns to that learned selector. That question does not arise in DYLAN's setting. The closest comparison in our experiments is a planner-fixed skill order, which we evaluate directly (the prescribed control, 69.38% on Mines). Alongside DYLAN, the revised Related Work now discusses hDQN and the distinction between recursive and hierarchical optimality in MAXQ and ALisp, in addition to the planner-guided methods already covered (PEORL, SDRL, Illanes et al., 2020, and reward machines).

The paper emphasizes the joint learning of the high-level option-selection policy (controller, not the planner) and low-level skill policies as a central contribution. However such simultaneous optimization of high- and low-level policies is established in hierarchical RL, including Option-Critic and SAC-X [2]. Thus, I would suggest the authors explain more explicitly the contribution and how it is distinct from previous works.

We agree that optimizing high- and low-level policies together is established, and we do not claim joint learning itself as the contribution. The contribution is how to credit skills that are trained on their own, heterogeneous rewards when control returns to a learned selector. Neither method addresses this. Option-Critic trains every option on the same task reward, learns the terminations, and bootstraps from the value of the policy over options after every termination. In PGOC each skill is trained on its own local reward, terminations are given by the specification, and a skill receives the controller's value only when it succeeds (Eq. 6); a failed execution receives none, so the extra credit is tied to task progress. SAC-X trains intentions on separate auxiliary rewards and learns a scheduler over them, but each intention optimizes only its own reward and receives no credit from the scheduler's value, and there are no preconditions restricting which intentions may run. Replacing PGOC's continuation target with the next option's value reduces Mines success from 62.80% to 23.46%, and removing the success gate reduces it to 56.38% (Table 21). We have made this contrast explicit in the revised Related Work.

AuthorsOpenReview ↗

Response to Reviewer JzE9 (Part 2)

Another question is that jointly learning the controller and skill may introduce a potential chicken-egg problem: the controller relies on sufficient skills to discover successful task sequences (the plan is provided externally) and estimate their continuation values, while the skills depend on the controller to select them and provide meaningful training.

This dependency exists in any jointly trained hierarchy, and PGOC's design addresses it directly. First, skills do not depend on the controller for a learning signal. Each skill is trained on its own local reward whenever it runs, and the controller's credit is an addition paid only on success. Second, the controller does not need to discover task sequences. The prerequisite masks admit only options whose preconditions hold, so early skills are selected and trained first and later options become admissible as earlier ones succeed. Third, the plan-progress rewards credit the controller for each requirement as it is met, not only on task completion. Removing the prerequisite mask collapses Mines success to 0.00% (Table 2), and the per-achievement curves in Appendix B.8 show the staged progression.

This leads to an issue: if an early skill fails to reach a required poststate, subsequent options may never become admissible or receive sufficient training. Consequently, the controller may struggle to learn meaningful selection policies for downstream options, while those options cannot improve without being executed. Such dependencies could lead to sparse learning signals and poor sample efficiency as the planning horizon and number of options increase, which constrains PGOC's capability.

SCALAR, whose specification format PGOC adopts, addresses this with two mechanisms. Pivotal Trajectory Analysis refines a skill's specification from its first successful trajectories, and Frontier Checkpointing restarts training from states where a skill's preconditions first hold, so downstream skills can train before upstream ones are reliable. PGOC deliberately uses neither. We hold the specifications fixed and train from fresh starts so that our comparisons vary only how skills are selected and credited, without a changing specification or training distribution as additional variables. Even so, PGOC reaches 62.80% Mines success from fresh starts, against 7.66% for SCALAR-Hybrid with its resets. Both mechanisms are compatible with PGOC, and combining them is a natural extension. The limit you describe remains. Our Limitations paragraph already notes that PGOC does not correct erroneous specifications during learning, and we have added the case you raise there: with fixed specifications, if an early skill never succeeds, the options that depend on it never become admissible and are never trained.

Thus, I would like to ask the authors how the option is designed?

The options are SCALAR's skill specifications and expert-sharing map, used unchanged and listed in Appendix I. Each option has an initiation predicate, a completion predicate and a local reward, all evaluated on the symbolic state; options with the same behavior, such as repeated gathering, share an actor-critic expert but remain separate controller choices. The main Craftax library has 13 options over 10 experts.

and what is the limit of PGOC in terms of scalability?

The libraries in the paper range from 8 options (Eat Plant) to 62 (ZombieHorde: approach, defeat, retreat and recover cycles for 16 zombies, over four experts). The number of options the controller faces at once stays small, because the masks admit only options whose preconditions hold and whose effects are still needed, so library size matters less than the raw count suggests. With 62 options on ZombieHorde, PGOC-Hybrid has the highest mean in Table 1 (13.03 kills), though its interval overlaps SCALAR's. We have not systematically varied plan length or library size, so we make no general scaling claim; the main limit we know of is the dependence on early skills described above.

Moreover, does the learned option's behavior align with its semantic meaning?

In terms of what it achieves, yes, by construction. An execution counts as successful only when the option's completion predicate holds, so every credited completion has produced the specified effect. How the option achieves it is learned. Figure 4 traces matched episodes, showing the gathering, crafting, exploration, combat and descent options in sequence. We will add per-option completion rates in the camera-ready version.

AuthorsOpenReview ↗

Response to Reviewer JzE9 (Part 3)

The assumptions and the problem definitions of PGOC are not clear: what assumptions does PGOC make, and what kind of problem does PGOC aim to address?

Thank you for raising this. The setting was spread across the abstract, Section 2 and Section 3.1. We have now gathered the assumptions into a single "Problem setting" paragraph at the start of Section 3 (lines 167–172).

The problem is the one named in the first sentence of the abstract (lines 9–11). A symbolic skill specification says when a skill may run, what it must achieve and how it is rewarded, but not how to learn the skills or how to choose among them. PGOC takes the specifications as given and addresses the second question. It learns the skills, and a controller that chooses among them, together from environment interaction (Section 2, lines 139–140 and 143–144).

PGOC makes four assumptions. First, each skill is supplied with a precondition, an effect and a reward function in SCALAR's format, written by hand or proposed by a language model. Second, these are evaluated on a symbolic description of the state. Section 3.1 (lines 176–181) defines three quantities: sts_t is the environment state, zt=ϕ(st)z_t=\phi(s_t) is its symbolic abstraction, and hth_t is the representation the policy network computes from the agent's observations. Preconditions, effects and skill rewards are evaluated on ztz_t, so ztz_t decides which skills may run, whether they have succeeded and what reward they receive. The controller and the skills choose actions from hth_t. Third, the task is given as required amounts of effects. Each requirement jj has a target amount BjB_j, which is the number of completions required when each successful execution produces one unit (Section 3.2, lines 214–217). Fourth, learning is online.

Within this setting, the paper's specific question is how a skill should be credited when control returns to the learned controller (Section 3.3).

Inadequate positioning leads to insufficient baseline comparison and ablation: for example, If we treat PGOC as an exploration paper, we are missing baseline exploration methods such as NovelD [3] and DEIR[4].

PGOC is not an exploration method. Its question is how to learn and select skills whose specifications are given, as now set out in the Problem setting paragraph of Section 3. NovelD and DEIR add a novelty bonus to the reward of a single policy and make no use of skill specifications. Our comparison against a learner without specifications is PPO with the same backbones; as Section 4 states, "PGOC versus PPO tests the supplied skill structure" (lines 356–357). A novelty bonus would complement PGOC rather than replace it, since it could be added to each skill's reward.

If we treat PGOC as a reward-shaping paper, we are missing reward machine [5] and Dylan [1] as comparisons.

The plan-progress reward is one component of PGOC rather than its subject. It is the controller's reward only; each skill keeps its own reward (Section 3.2). In Table 2, replacing it with the raw task reward lowers Mines success from 62.80% to 54.26%. Section 2 contrasts it with reward machines (lines 124–129). Both track progress toward the task, but a reward machine fixes the order of subtasks in its automaton, whereas PGOC's controller learns the order. As in our reply to Reviewer SUzg, compute constraints prevented us from running a reward-machine baseline during the discussion period. We will include one in the camera-ready, with the same specifications and discount, and do not expect it to change our conclusions. As discussed in our answer above, DYLAN shapes the reward of a single flat policy and trains no separate skills.

Then, for option reuse, we lack compositional generation methods such as DiffStitch [6] and TMD [7].

Thank you for these references. Both study composition in offline RL, where a policy is learned from a fixed dataset. DiffStitch augments the dataset with generated transitions that connect low-reward trajectories to high-reward ones. TMD learns a quasimetric representation from offline data, whose triangle-inequality structure lets a single goal-conditioned policy reach goals by combining pieces of different trajectories. PGOC learns online, from environment interaction with no dataset (item (iv) of the new Problem setting paragraph). Neither method could be applied here without first collecting a dataset, which would change the problem. The composition is also of a different kind. PGOC's controller chooses among separately specified skills, each trained on its own reward. DiffStitch and TMD recombine segments of logged trajectories, using a single policy and no skills.

AuthorsOpenReview ↗

Response to Reviewer JzE9 (Part 4)

And how about comparison with hierarchical methods such as hDQN [8], option critic, and feudal networks, and in general, goal-conditioned RL?

Section 2 discusses these methods (lines 98–120), and we have added h-DQN there. They differ from PGOC mainly in what the learner is given. Option-Critic learns its options from the task reward alone, FeUdal Networks and HIRO set goals in a learned embedding or in state space, and goal-conditioned RL in general trains a single policy conditioned on a goal, typically a target state or its embedding.

PGOC instead receives skills specified by preconditions, effects and rewards over a symbolic abstraction. h-DQN is the closest, since its subgoals are also supplied. Its meta-controller, however, has no prerequisites restricting its choices, and its low-level controller is rewarded only for reaching the chosen subgoal.

For option-critic specifically, we test its termination backup inside PGOC with everything else held fixed. Bootstrapping each option from the controller value after every termination, as option-critic does, gives 56.38% Mines success against 62.80% with PGOC's success gate (Table 21, Appendix K.1). As committed in our reply to Reviewer SUzg, we will add a standalone option-critic baseline with the same initiation sets and rewards in the camera-ready.

Furthermore, how is the reward designed to obtain the desired high-level policy and low-level option?

Each level has its own reward. Each skill is trained on the reward in its specification. Appendix I lists the skills, and Appendix G gives their Craftax instantiation. PGOC and SCALAR use identical skill specifications, so the comparison between them holds these rewards fixed (Section 4, lines 356–357).

The controller is rewarded for progress toward the task. Each requirement jj has a target amount BjB_j. The controller earns credit for each unit of progress toward it, whichever skill produced that progress, until the target is reached. Credit is capped at the amount still outstanding, so the total per episode is bounded and repeating a skill earns nothing once its requirement is met (Section 3.2, Eqs. 2–3, lines 214–229).

The two rewards meet at skill boundaries. When a skill succeeds, it also receives the controller reward earned during its execution, plus the controller's estimate of the value that follows (Section 3.3, Eq. 6, lines 261–264). A skill is thus credited for what its success enables, and only when it succeeds.

and how brittle is the reward design of the pipeline?

Table 2 tests this by replacing the controller's plan-progress reward with the raw task reward on every task, with everything else held fixed. Plan progress does better on the Craftax Ladder and Mines milestones (80.17% vs 69.20% and 62.80% vs 54.26%), and the two are nearly equal on Dungeon, Diamond and TreasureDash. Quest-Hard favors plan progress (64.41% vs 29.33%), though with wide confidence intervals. The exception is Craftax-Classic Eat Plant, where plan progress does much worse (16.12% vs 77.35%). Section 4.3 traces this to the task (lines 418–426). The crop ripens only after about 600 game steps, and the plan-progress reward is exhausted once the preparation is done, so it gives no signal while the agent waits. The paper reports this as a limit of the construction, which suits goals with incremental, immediate progress.

Will PGOC learn shortcuts or fail to learn if multiple paths exist?

PGOC is built for tasks with multiple paths. The controller is credited for progress toward a requirement whichever skill produced it (Section 3.2). It chooses among the admissible skills from its observations, so it can take whichever path the current state favors. Randomized TreasureDash tests this. Its floor geometry is drawn per episode, so no single order is best. The learned controller scores 22.21 there, against 10.88 for the prescribed order and 13.31 for uniform selection over the same admissible skills (Section 4.4, lines 485–492).

As for shortcuts, the controller's reward cannot be farmed. Progress credit is capped at the amount each requirement still needs. Once a requirement is met, the continued-need mask removes the options that serve it (Eq. 1). A skill receives the controller's continuation value only when its success predicate holds, so it cannot collect that credit by ending early. What PGOC does not guard against is an error in the specification itself. If a success predicate can be satisfied without the intended effect, PGOC will learn to satisfy it. The Limitations paragraph states that PGOC does not correct erroneous specifications during learning (lines 506–510).

Reviewer_SUzgOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

The paper studies how to learn and combine skills when each skill comes with a symbolic specification: when it can start, what counts as success, and its reward. These specifications come from SCALAR (Zabounidis et al., 2026), which runs skills in a fixed order and credits each skill using the next skill's value.

PGOC instead trains a high-level controller and the skills together. It uses:

  • Masks that block skills whose preconditions fail or whose job is already done.
  • Capped plan-progress rewards for the controller.
  • A controller-value bootstrap: a skill that succeeds bootstraps from the controller’s value at the new state.

On Craftax, PGOC reaches the Gnomish Mines in about 63% of episodes, versus about 8% for SCALAR and at most 4% for PPO.

The authors conclude that their credit assignment drives the Craftax result, while learning the order helps only when the best order depends on the state.

Critical strengths
  • Large, carefully measured gain over SCALAR on a hard long-horizon task. The evaluation uses frozen policies, 10,240 evaluation episodes per seed, and per-seed reporting.
  • Thorough ablations and controls. These include a discount/reset factorial and a frozen-skill control that isolates the selector.
  • Honest disclosure of limitations. The authors report that model selection used the test seeds, that empty option sets occur, and that the method fails on Eat Plant.
Critical weaknesses
  • Limited novelty. The core components are standard: The masks are initiation sets from the options framework (Sutton et al., 1999). The controller-value bootstrap is option-critic’s termination backup, which uses V_Ω(s’) when an option ends (Bacon et al., 2017). It is also close to MAXQ’s completion function (Dietterich, 2000). Plan-progress rewards are subgoal rewards from symbolic structure, as in reward machines (Toro Icarte et al., 2022) and planner-guided HRL (Illanes et al., 2020). The genuinely new pieces, success gating and positive clipping, matter little in the ablations (56.4% and 62.2% without them, versus 62.8%). The paper reads more as a fix to SCALAR’s credit assignment than as a new method.

  • The learned option critic does not help on the main task. On Craftax, a fixed order and uniform random selection both beat it. The case for learned selection rests on one small task, TreasureDash. There, the fixed-order baseline cannot use the layout at all, so it is a weak comparison.

  • Key baselines are missing. There is no comparison to option-critic or SMDP option learning with the same initiation sets and rewards, and none to reward-machine or planner-guided HRL methods. SOL and HBS do not use the specifications, so their near-zero scores do not test PGOC’s design.

Questions for authors
  • How does Eq. 6 differ from option-critic’s termination backup beyond success gating and clipping?
  • How do option-critic and a reward-machine baseline perform with the same initiation sets, rewards, and γ = 0.999?
  • Can you report SCALAR with all learner settings matched, and PGOC on new seeds with the configuration locked?
  • Since uniform selection beats the learned controller on Craftax, what does the controller contribute there?
  • On TreasureDash, how does a plan chosen after seeing the layout compare to the learned controller?
  • Why does PGOC fall well behind SCALAR on Quest-Hard and Eat Plant?
3 comments
AuthorsOpenReview ↗

Response to Reviewer SUzg (Part 1)

Thank you for a careful and precise review, and for recognizing the size of the gain over SCALAR, the thoroughness of the ablations and controls, and our disclosure of limitations. Your main concerns are how PGOC differs from option-critic, MAXQ and reward machines, that the learned selector does not help on Craftax, and that option-critic and reward-machine baselines are missing. In response to your review, the revised Related Work now distinguishes recursive from hierarchical optimality in MAXQ and ALisp (lines 99–102) and compares how Option-Critic, SAC-X, SOL and PGOC credit their low-level policies (lines 115–120). We agree with several of your observations, including the small effect of clipping and that the evidence for learned selection rests on one task.

The controller-value bootstrap is option-critic's termination backup, which uses V_Ω(s') when an option ends (Bacon et al., 2017).

  1. How does Eq. 6 differ from option-critic's termination backup beyond success gating and clipping?

Option-critic trains every level on one task reward, so the option's value and VΩV_\Omega estimate the same return and bootstrapping across a termination is consistent by construction. PGOC's skills each have their own specified reward, so no return is shared. A skill's critic estimates its local return up to the boundary, and the method must decide what the skill is credited with afterwards. Bootstrapping from the next skill's value imports that skill's reward definition, a different objective on a different scale; with only this target changed, Mines success falls from 62.80% to 23.46% (Table 2). PGOC instead bootstraps from the controller's value, the one objective common to all skills, and on success pays the skill the controller reward earned during its execution, so every skill's continuation is measured against the same plan-progress return.

The boundary term coincides with option-critic's at a successful termination; the problem it solves, composing options with heterogeneous rewards, does not arise in option-critic. PGOC also differs in what is supplied rather than learned. Its terminations are given as predicates, and selection is restricted to admissible options.

It is also close to MAXQ's completion function (Dietterich, 2000).

The completion function is closest in form, but in MAXQ it belongs to the parent: it values finishing the parent task after a subtask returns and does not enter the subtask's own learning, so each subtask is optimized for its own goal regardless of the context that invoked it (recursive optimality). PGOC passes the controller's value into each successful skill's return, the role ALisp's external value term plays in recovering hierarchical optimality (Andre & Russell, 2002). Thank you for this connection; we have added the distinction to the revised Related Work (lines 99–102).

The genuinely new pieces, success gating and positive clipping, matter little in the ablations (56.4% and 62.2% without them, versus 62.8%).

The success gate makes this credit conditional. Option-critic bootstraps from VΩV_\Omega after every termination, whereas PGOC gives a skill continuation value and controller reward only when its completion predicate holds, and nothing otherwise. A skill therefore cannot earn credit by ending in a state the controller values without producing its specified effect. The extra credit is a reward for success, and clipping ensures success is never valued below failure. Removing the gate lowers Mines success from 62.80% to 56.38% (Table 21).

We agree that the clipping alone is negligible (62.16% without it vs 62.80%).

AuthorsOpenReview ↗

Response to Reviewer SUzg (Part 2)

The masks are initiation sets from the options framework (Sutton et al., 1999). [...] Plan-progress rewards are subgoal rewards from symbolic structure, as in reward machines (Toro Icarte et al., 2022) and planner-guided HRL (Illanes et al., 2020).

The masks are initiation sets in the sense of the options framework. What PGOC adds is deriving them from the skill specifications and combining them with continued-need predicates, which close an option once the requirements it serves are met (Eq. 1). Removing the prerequisite mask drops PGOC-Hybrid from 62.80% to 0.00% Mines success (lines 400–403, Table 2). The plan-progress reward differs from a reward machine in role and structure. It is the controller's reward only; skills keep their own rewards and receive progress credit only through success-gated payment. It credits capped effect counts in any order, whereas a reward machine fixes the order in its transitions. Crediting Craftax's counted requirements (15 wood, 9 stone, 16 arrows, 8 defeats, among others) in any order would need a machine state for every combination of counts, where PGOC keeps one counter per requirement. The closest reward-machine method, HRM, also learns options with a high-level policy, but trains each option only on its own subgoal. Our closest control removes continuation credit, so a skill keeps only its own reward and the controller reward earned during its execution; it reaches 42.05% Mines success against 62.80% for PGOC (Table 2), so the boundary credit is what connects a skill to what follows it.

The learned option critic does not help on the main task. On Craftax, a fixed order and uniform random selection both beat it. The case for learned selection rests on one small task, TreasureDash. There, the fixed-order baseline cannot use the layout at all, so it is a weak comparison.

We agree with the facts, and we should have discussed them more directly. With initiation and continued-need masks, every sequence of admissible options is a valid plan, so even uniform selection only chooses among plans a planner could have produced. On Craftax, prescribed and uniform selection lie within each other's 95% intervals (69.38 ± 1.48 vs 67.91 ± 3.65), so these plans are worth about the same. There is little to gain from choosing among them, and learning the choice has a cost, because under a learned selector the controller value that every skill bootstraps from keeps changing as the selector trains. Consistent with this, prescribing the order lowers option-value loss and within-option advantage variance in all five seeds (Figure 3). Learning pays only when the best order cannot be fixed before interaction. On randomized TreasureDash the layout is drawn per episode, so the prescribed order, the best a planner can commit to in advance, cannot exploit it. There the learned controller scores 22.21 against 10.88 for the prescribed order and 13.31 for uniform. The fixed order is what a planner can achieve without observing the state, so it is the relevant baseline here. We agree that one task is thin evidence for this side of the trade-off.

Key baselines are missing. There is no comparison to option-critic or SMDP option learning with the same initiation sets and rewards, and none to reward-machine or planner-guided HRL methods. SOL and HBS do not use the specifications, so their near-zero scores do not test PGOC's design.

  1. How do option-critic and a reward-machine baseline perform with the same initiation sets, rewards, and γ = 0.999?

The paper tests the design choices these methods differ in as controlled swaps within the same system. Option-critic's termination backup, which bootstraps from the controller value after every termination, is the no-success-gate row (56.38% vs 62.80%). Chaining to the next option's value, as in SMDP-style option sequencing, is the next-option row (23.46%). Training each option only on its own reward, as in hierarchical reward-machine methods, is the no-continuation row (42.05%). The raw task reward in place of plan progress gives 54.26%. Planner-guided methods such as SDRL, PEORL and Illanes et al. (2020) execute options in a planner-determined order. Our prescribed-order control is that design with PGOC's credit assignment (69.38%), and the SCALAR-style rows of Table 21 are that design with next-skill credit (12.77%). SOL and HBS are included because they are the published hierarchical methods evaluated on these benchmarks; we agree they do not use the specifications and do not test PGOC's design.

Because of compute constraints we could not run standalone option-critic and reward-machine baselines during the discussion period. We will include both, with the same initiation sets, rewards and γ=0.999\gamma=0.999, in the camera-ready version. We do not expect them to change the conclusions, since the controlled swaps above already isolate the components those methods differ in.

AuthorsOpenReview ↗

Response to Reviewer SUzg (Part 3)

  1. Can you report SCALAR with all learner settings matched, and PGOC on new seeds with the configuration locked?

In the matched comparison of Table 21, PGOC with SCALAR's estimator, sharing our learner settings, masks, rewards, γ=0.999\gamma=0.999 and fresh starts, reaches 12.77% (prescribed) and 6.12% (learned); the SCALAR code at γ=0.999\gamma=0.999 reaches 3.60% with practice resets and 1.28% without, below the released 7.66%. The configuration selection on these seeds, disclosed in Appendix B.5, concerned only two controller settings, the entropy coefficient and the readiness features, and moved Mines success by about 3–4 points (Table 6). The untuned base configuration already reaches 61.53% over three seeds, against 7.66% for SCALAR, so selection cannot account for the gap. The core components (masks, boundary target, plan-progress reward) were not selected on these seeds. The prescribed and uniform controls reuse the configuration selected for the learned controller, so any bias favors the learned controller, which they nonetheless outperform. We will report the released configuration on fresh seeds in the camera-ready version, but given these margins we do not expect any conclusion to change.

  1. Since uniform selection beats the learned controller on Craftax, what does the controller contribute there?

On Craftax, the controller's selection policy contributes nothing to final success (see our answer on the learned option critic above). Every skill bootstraps from its value function at a successful boundary, and the prescribed and uniform controls keep that value function. Removing that continuation credit lowers Mines success from 62.80% to 42.05% (Table 2). With the order prescribed, the final option-value loss is 13.1% lower and the within-option advantage variance 12.7% lower than with the learned selector, in all five seeds (Figure 3), so learning the selection policy on top of it has a measurable cost. This is consistent with a changing selector making the bootstrap target noisier, and on Craftax nothing is gained in return.

  1. On TreasureDash, how does a plan chosen after seeing the layout compare to the learned controller?

A plan chosen after seeing the layout attains the per-layout optimum, and the learned controller's mean is about 3.1 points below it (Section 4.4), so such a plan would do better and headroom remains. The prescribed order is instead the best a planner can commit to before seeing the layout, because the layout is drawn per episode.

  1. Why does PGOC fall well behind SCALAR on Quest-Hard and Eat Plant?

Eat Plant shows where the controller-value target fails. The target is only as informative as the controller's objective, and the plan-progress reward is exhausted once preparation completes, so through the roughly 600-step ripening wait the controller value carries no signal about surviving to eat the crop. Frozen policies plant in over 99% of episodes and die before the crop ripens (Section 4.3). The next option's value, by contrast, includes the Eat Plant skill's survival shaping (vital and death terms), which the plan-progress return does not. Targets that still carry this signal do much better, with 50.62% for the next option's value and 77.35% for the raw task reward. On Craftax Mines, where progress continues to the end of the plan, the ordering reverses (62.80% vs 23.46%). The choice of continuation target should therefore follow where the objective's signal lies. The Limitations paragraph states this case (lines 508–509). Quest-Hard results vary widely across seeds. PGOC-GRU (7.64%) and PGOC-FC (0.00%) fall well behind SCALAR, while PGOC-Hybrid's mean (64.41%) is above SCALAR-RNN's (56.02%), with 95% intervals of ±52 and ±31 points. Neither direction is supported by these intervals, and we will note this in the Limitations paragraph.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Global Reply (to all reviewers)

We thank the reviewers for their careful reading. Reviewer SUzg found the gain over SCALAR large and carefully measured and the ablations thorough, Reviewer JzE9 found the idea interesting and the results promising, and Reviewer bLVt noted PGOC's improvement over SCALAR on difficult tasks. We have uploaded a revised PDF with every change in blue, and the replies to each reviewer cite its line numbers. Here we summarize our answers to the concerns raised by more than one reviewer.

PGOC learns skills that each come with their own specified initiation set, completion predicate and reward, together with a controller that chooses among them. On Craftax, PGOC with recurrent backbones reaches the Gnomish Mines in 62.8–64.5% of frozen evaluation episodes, against 7.7–9.3% for SCALAR with the same skill specifications and at most 4.3% for PPO, over five training seeds with 10,240 evaluation episodes each (Table 1).

Reviewers SUzg and JzE9 asked how PGOC differs from option-critic, MAXQ, reward machines, SAC-X and DYLAN. In option-critic every option is trained on the same task reward, so bootstrapping from the policy over options after a termination is consistent by construction. When each skill has its own reward, a method must decide what a skill's return bootstraps from once control passes back to the controller, and bootstrapping from the next skill's value imports a different objective on a different scale. PGOC bootstraps from the controller's value, the one objective common to all skills, and only when the skill succeeds. MAXQ keeps this continuation value in the parent's completion function, outside the subtask's own learning, and PGOC's choice plays the role of ALisp's external value term instead. SAC-X, SOL and hierarchical reward machines train each low-level policy only on its own reward, and DYLAN trains a single flat policy with no separate skills. The revised Related Work makes these contrasts explicit (lines 99–102, 105–106, 115–120 and 124–125).

The paper tests the closest prior designs inside PGOC, holding the skill library, masks, plan-progress reward, learner settings, discount and budget fixed (PGOC-Hybrid, Craftax Mines success over five seeds; Tables 2 and 21).

Skill credit at the boundaryClosest prior designMines (%)
Controller value, on success onlyPGOC62.80
Controller value after every terminationOption-critic's termination backup56.38
No continuation valueSAC-X, SOL, hierarchical reward machines42.05
Next option's valueSMDP-style option sequencing23.46
Next-skill bootstrap with propagated advantageSCALAR's estimator6.12

With the order fixed in advance, as in planner-guided HRL, PGOC's credit reaches 69.38% and SCALAR's estimator 12.77%. Removing the positive-part clipping changes little (62.16%).

Reviewer SUzg asked what the learned controller contributes, given that it does not beat prescribed or uniform selection on Craftax. With prerequisite and continued-need masks, every sequence of admissible options is a valid plan. On Craftax, prescribed selection (69.38%) and uniform selection among admissible options (67.91%) lie within each other's 95% intervals, so these plans are worth about the same (Section 4.4). Learning the choice still has a cost, because the controller value that every skill bootstraps from keeps changing as the selector trains. Prescribing the order lowers the option-value loss by 13.1% and the within-option advantage variance by 12.7%, in all five seeds (Figure 3). Learned selection pays off when the best order depends on the state. On randomized TreasureDash it scores 22.21, against 10.88 for the prescribed order and 13.31 for uniform selection.

Reviewers bLVt and JzE9 found the method hard to follow and its assumptions unclear. The revised PDF adds a Problem setting paragraph stating PGOC's assumptions (lines 167–172); definitions of hth_t and ztz_t where they are first used (lines 176–179); a definition of rtor^o_t and of when an execution ends (lines 186–194); a limitation on skills that never succeed (line 507); Appendix L, which describes SCALAR's specification format with a worked example; and Appendix M, which describes the benchmarks and why they were chosen. Compute constraints prevented new experiments during the discussion period. For the camera-ready version we will add standalone option-critic and reward-machine baselines with the same initiation sets, rewards and γ=0.999\gamma=0.999, and the main configuration retrained on fresh seeds, so that the reported seeds are not the ones used to select it (Appendix B.5). Because the comparisons above already isolate the components these methods differ in, we do not expect them to change the paper's conclusions.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.