acceptodds
Under review as a conference paper at ICLR 2027

Going Beyond State-Reaching: Learning Abstractions for Intrinsically Motivated Option Discovery

Abstract

Temporal abstraction via options can improve exploration in vast environments. However, existing option discovery algorithms find subgoals that target all aspects of the state simultaneously. This state-reaching approach produces options that only apply in narrow regions of the state-space, eventually causing an explosion in the number of options that overwhelms the agent, and impedes progress on its primary task of reward maximization. We introduce an algorithm that instead identifies a small, relevant subset of features for each subgoal, yielding options that generalize broadly and accelerate exploration. Our approach learns abstract, transferrable options and achieves rapid exploration in three sparse-reward, image-based domains, including the Atari game MontezumasRevenge.

Primary area:
reinforcement learning
Keywords:
Reinforcement learning, Hierarchical Reinforcement Learning, Exploration

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_9ETfOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper presents an approach for learning options and option policies for sparse reward RL problems. The core idea is to use novelty in observed state sequences as a surrogate for detecting subgoal candidates, and to then use image-based feature extractors to identify parts of images that contribute the most to their corresponding states' novelty. Shapley value based credit assignment is then used to identify relevant features and subgoals are created for achieving salient feature values. The central claim of the paper is that unlike related work, this work defines subgoals based on features rather than full states.

While the ideas presented are interesting and result in feature-based options that are potentially more decomposable and reusable than many other approaches for learning options, some of the claims of novelty are difficult to support.

Critical strengths
  • Discovery of salient features and feature based subgoals has the potential to support greater transfer across problem instances.

  • Good empirical analysis on three test domains

  • Paper is well written, with ideas that are clearly motivated

Critical weaknesses
  • Transfer performance is not evaluated quantiatively

  • Claims that the presented approach transfers seamlessly to factored state representations are not supported by theoretical or empirical evidence. In particular it is not clear that the presented approaches for feature discovery would work equally well in factored domains, and recent approaches in that direction (e.g. Nayyar et al., AAAI 2025) don't use each dimension as a candidate feature. Empirical comparison with such approaches would help justify these claims.

  • It is difficult to argue that this is the first work that goes beyond treating full states as subgoals. Prior work has considered more general notions of subgoals. E.g., Bagaria et al. (IJCAI 2021) define option termination conditions using classifiers; other examples include Veeriah et al., NeurIPS 2021. Much of the work on learning subgoals in latent spaces implicitly achieves the desired concept of defining subgoals in terms of sets of states rather than single states; the difference here could be argued to be not merely going beyond state-reaching, but in terms of learning decomposable subgoal features that can be recomposed in novel ways for novel reward functions. Such a claim would be more justifiable.

Bagaria et al. "Robustly learning composable options in deep reinforcement learning." IJCAI 2021.

Veeriah et al. "Discovery of options via meta-learned subgoals. NeurIPS 2021.

Nayyar et al. "Autonomous option invention for continual hierarchical reinforcement learning and planning." AAAI 2025.

Questions for authors
  1. Are the interactions for learning subgoals, intra-option policies and inter-option policies included in Fig. 5?

  2. The phases mentioned in the previous question are cleanly separated as different modules in the overall algorithm. Can you provide a breakdown of the number of interaction steps required for the components listed above?

  3. Have you tried to implement and evaluate this approach for transfer across problem instances? It seems that transfer performance would be a better metric for distinguishing your approach from existing work rather than merely learning options that go beyond state-reaching.

  4. Can you provide data justifying the additional cost of computation required for feature discovery, pruning, and learning inter and intra-option policies vs flat RL? E.g., clock-time based learning curves including including all the steps required for the presented approach.

5 comments

We thank the reviewer for a careful review. We added transfer experiments (Appendices E and G.2), two factored-state domains with the extraction step removed (Appendix G). Responses to each point follow.

Questions

  1. Are the interactions for learning subgoals, intra-option policies and inter-option policies included in Fig. 5? Yes, all of them. The method is fully online: subgoal discovery, intra-option policy learning and the policy over options run inside a single interaction loop (Algorithm 1), with no separate phases, so the x-axis of Fig. 5 counts every environment frame the agent experiences. The only experiments that use a separate discovery-only stage are the ablations in D.2 and D.3, where it serves to hold the subgoal set fixed across methods, and this is explicitly stated there.

  2. Breakdown of interaction steps per component. The components are separate modules in code but not separate phases in time, so there is no per-component interaction budget; every frame is used by all of them. Attribution and classifier construction operate on stored frames and consume no interaction; intra-option policies train off-policy from all frames via hindsight relabeling; the policy over options combines predictions from CFN and the goal-conditioned value function, so it also does not consume frames of its own.

  3. Show policy transfer across problem instances. Yes, this was a major addition in the revision:

    • Montezuma’s Revenge (Appendix E, Fig. 13): two options discovered in room 1 reach their subgoals when executed from rooms 2 and 6 under very different conditions like number of lives, game score, and enemy locations .
    • Ant U-Maze, same option, new contexts (Appendix G.2, Fig. 18): a discovered "posture option"---i.e., an option that target a particular joint angle configuration of the Ant---reaches its subgoal from four distant maze cells with random headings and joint angles in 399 of 400 attempts.
    • Ant U-Maze, new goals (Fig. 19): given target locations never used in training, the agent composes its discovered options to reach them at a 90–100% rate, whereas conditioning the same policy directly on the target fails for far-away goals.
  4. Computational cost relative to flat RL. Measured on identical hardware, Montezuma's Revenge runs take 1.25x the wall-clock of flat R2D2 for the same number of frames, and use two GPUs rather than one: the second hosts the CFN novelty model and the image segmenter. Whether or not this additional cost is justified depends on the application, we can include a note on computational requirements for the camera-ready if that helps. We have added Appendix H to answer this question; Fig 21 compares the different methods on state-based taxi in terms of wall-clock time.

Weaknesses

  1. The claim that the approach transfers seamlessly to factored state representations is unsupported. We ran our method on two factored-state domains with the extraction step deleted (Appendix G): in state-based Taxi, the ordering of methods matches the image-based domains and our agent reaches 0.95 success in 2.5x fewer frames than CFN (G.1). In Ant U-Maze, each of the 29 state dimensions is a candidate feature, the agent discovers that xy is the relevant subspace, and it matches final performance of an agent given the oracle xy goal space (G.2).

  2. Comparison to Nayyar et al, 2025. We thank the reviewer for the pointer to Nayyar & Srivastava (2025), which we had missed. It is relevant and we will discuss it in Section 2. Its setting differs from ours in inputs (a given set of state variables and a simulator, versus pixels) and objective (emphasis on continual multi-task planning, versus online single-task exploration), so we do not include a direct comparison. We hope that the comparison to Deep Skill Graphs with privileged access to the goal space will be a sufficient addition.

  3. Related work that also learns subgoal classifiers. On the cited work: the termination classifiers of Bagaria et al. (2021) are learned over a designer-chosen goal space (the agent’s xy position). The oracle-goal-space agent in Appendix G.2 of the revision is that idea instantiated in Ant U-Maze, and our agent, which must discover the goal space, matches its final success rate. Veeriah et al. (2021) take the full state as input; such methods could in principle discover factored subgoals, but end-to-end representations in RL tend to be entangled (Rodriguez-Sanchez et al., 2025; Kumar et al., 2025), and the factored subgoals in our Figs. [13, 16] show that our method finds them in practice.

We would be glad to clarify anything further during the discussion period.

Today is the last day of the discussion period, so we wanted to check in to see if there’s anything else you would like addressed; we believe we have cleared up all your concerns.

Thank you for responding to the concerns I raised in my review. I think the shift towards evaluating generalization and transfer is the right approach, however, it requires substantial new analysis and a clear, well formulated empirical evaluation. In particular, it appears that this approach applies more to navigation tasks than to more general settings. For instance, in Montezuma's revenge the learned subgoal highlighted in Appendix E highlights a generic subgoal (reach "Center left" of screen). Such goals are in fact too non-discriminative: they would increase the effective branching factor since it is not clear when they will be useful and it is not clear how they could be eliminated. In addition, the extra GPU/computational cost needs to be considered carefully and reported with the experiments for all domains. Finally, there are several approaches for discovering options in static as well as continual RL settings with state based input or factored representations. Pointers for these methods can be found in the papers discussed above.

I encourage the authors to continue developing this thread of research. However, I also believe that a version addressing these concerns requires more extensive changes and a full review. I will therefore maintain my current score.

Reviewer_fvQTOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper propose a method for intrinsic motivation based option discovery, and it particularly emphass on learning abstract subgoals instead of a simple state-reachign subgoals. This direction is definitely worth further investigating. The research community has seen the limitation of goal-reaching options for a long time. So this is an interesting paper to read.

We know that, the difficulty in this research field is how can you define "useful" latent features as "useful" subgoals so that we can discover "useful" option. This paper, essentially, what is new here is that they use novelty detection and feature attribution to identify the relevant features of a novel state, and use these features to define subgoals for option learning. This newly proposed approach is not surprising though, still worth exploring. I appreciate the authors' efforts in this direction.

And lastly, they hope these abstract options can generalize across different states and improve exploration. Experiments on three environments, including Montezuma's Revenge, have shown some promising results, but with limited baselines and not enough evidence to fully demonstrate the advantages of the learned options.

Critical strengths
  • the paper is well-written, I especially like the motivation here: learning options based on relevant features rather than the entire state!
  • this motivation makes sense for learnign resuable skills, and the authors explain this problem quite well! I feel excited to read this part!
  • the proposed method is simple and intuitive: just using feature atrriibution to identify what makes a state novel. I think it's a reasonable direction for option discovery.
  • experiment results are promising, especialy on Montezuma's Revenge (which we know are difficult and a good test bed for option discovery).
  • overall, I think this paper can do better if the authors can do more comprehensive experiments other than the listed three: VisualTaxi, MiniGrid KeyCorridor, Montezuma's Revenge. And the baselines seem not strong enough. If give more time in the experiment part, this can be good paper for publication in ICLR.
  • BTW, I really like reading your related works section! Compared to other two papers I am assigned (I'm not sure how these were assigned to me) in embodied AI, reading their related works section is a terrible experience for me.
Critical weaknesses

Well, you have read my previous section, you can imagine how I will write the critical weakness.

Here are the detailed reviews, with the hope to help the authors to iterate the paper for a better version.

1). It's a good paper! If the authors can do more comprehensive experiments, I will raise the score.

  • The authors evaluate the method on three environments. Montezuma's Revenge is certainly a difficult exploration problem, but I don't think these experiments fully demonstrate the advantages of the proposed abstract option discovery method. In particular, the authors claim that their method can learn more generalizable and reusable options, but most of the evidence is based on showing that the subgoal classifiers can be triggered in different states. This does not necessarily mean the learned option policies can generalize well. I would like to see more direct experiments on option reuse and transfer, for example, whether an option learned in one environment configuration can still successfully achieve its subgoal in a different configuration. This would provide much stronger evidence for the main claim of the paper.

2). Baselines seem relatively weak!

There has been extensive research on intrinsic motivation and option discovery, and I think the paper needs comparisons with more established methods in this area. For example, Proto-goal RL (Bagaria, 2023 from Tom Schaul's group) and other feature-based option discovery approaches are closely related to the proposed method. Even if these methods cannot be directly applied to image observations, the authors could evaluate them using the same extracted features, making the comparison more fair. This would also help clarify how much improvement actually comes from the proposed feature attribution method, rather than simply using a better subgoal representation.

Very good paper! Very promising and simple method. Just need more time to work on the experiment part! Compared to other embodied AI papers I am reviewing, I enjoyed a lot of happy hours reading this one! That's the benefits of being a reviewer I guess.

Questions for authors

check my questions about exps and baselines above.

5 comments

We thank the reviewer for deeply engaging with our paper. We hope that our new experimental results address their remaining concerns.

  1. Existing evidence suggests subgoal classifiers that can transfer, not option policies. We agree this was a gap, and we added direct tests of the kind the reviewer describes: an option learned in one configuration, executed in another.
  • Montezuma’s Revenge (Appendix E, Fig. 13): two options discovered in room 1 reach their subgoals when executed from rooms 2 and 6, with different layouts, lives, scores and enemy positions.
  • Ant U-Maze (Appendix G.2, new domain): a discovered posture option (targeting joint angles), executed from four distant maze locations with random headings and joint angles, reaches its subgoal in 399 of 400 attempts (Fig. 18). Further, given goals never used in training, the agent composes its discovered options to reach them at a 90–100% success rate, where conditioning the same policy directly on the target fails for distant goals (Fig. 19b).
  1. The paper needs comparisons with more established methods in this area. We added one established method and a stronger version of our state-reaching baseline:
  • Deep Skill Graphs (Appendix G.2): the closest prior option discovery method for exploration, given the oracle xy goal space from its original evaluation in Ant U-Maze. Our agent, which must discover the goal space, matches its final success rate (≈0.8) at ~3x the samples; flat TD3+CFN is unable to reliably solve the task.
  • State Similarity (Section 4.4, Fig. 5, all three domains): subgoal achievement over the full state, with the threshold tuned per domain for best final return (Table 5). It improves on State Equality everywhere and matches our method in VisualTaxi, but stays well below it in KeyCorridor and Montezuma’s Revenge.

Responses to additional comments:

  1. On Proto-goal RL. We see our method as a learned version of Proto-goal RL. That method assumes a large space of candidate goal predicates is given and prunes it using observability, reachability, controllability, novelty and task-relevance. Our agent constructs the predicates from experience: each is built from a state the agent actually reached, so it is observable by construction, and attribution selects the relevant feature combination from a single novelty event rather than enumerating combinations and pruning them. Novelty, reachability, and task relevance play similar roles in both methods. The one Proto-goal RL criterion we do not yet apply is controllability: some discovered subgoals attend to the skull’s position, which the agent cannot control, and that criterion would prune them; we note this in the limitations. Rather than compare to Proto-goal RL directly, we compare in Ant U-Maze against DSG given the oracle goal space (Appendix G.2); since Proto-goal RL’s output is a pruned goal space, an agent handed the ideal goal space upper-bounds what it could provide.
  2. How much improvement comes from the feature attribution method, rather than from a better subgoal representation. Our Ant U-Maze experiments provide an answer: the oracle agent and ours use the same feature-scoped subgoals and differ only in who selects the features—a designer or feature attribution. Any selection method, Proto-goal RL included, can at most recover the oracle’s goal space, so the gap between the two curves bounds what a better selector could gain: attribution reaches the oracle’s final success rate at roughly 3× the samples, and selects the xy features first (Fig. 20). The separate question of whether feature-scoped subgoals beat full-state ones is answered in Appendix D.2 (≈1,100 vs. 0 with identical subgoals) and Appendix G.1 (full-state classifiers are no better than flat R2D2 in state-based Taxi).

In total, the revision adds two domains, two baselines, two transfer tests and three ablations. We hope these are the comprehensive experiments the reviewer had in mind.

Thank you authors! The new option-transfer results already address one of my main concerns, and the oracle-x-y DSG comparison is a useful reference for evaluating whether the method can discover an effective goal space.

In your response, you mentioned "since Proto-goal RL’s output is a pruned goal space, an agent handed the ideal goal space upper-bounds what it could provide."

  • I do not agree with the "upper-bounds" description.
  • My main concern about the baselines remains.
  • I requested a comparison with Proto-goal RL or another feature-based goal discovery method using the same input features. The oracle-x-y DSG comparison is providing an oracle goal space, I do not think it can establish a performance upper bound for Proto-goal RL, which also involves goal filtering and selection.
  • I would therefore like to see a direct comparison with Proto-goal RL on State Taxi, using the same state features and matching the reward information.

Your response was posted on November 16, and the discussion DDL is Nov 18. This makes adding more baseline comparison more difficult.

I understand that, I appreciate the substantial additional work. But next time, pls be careful about selecting baselines that can examine your main contribution. I will raise my score to 3 weak accept. Again, still a good work.

AuthorsOpenReview ↗

Comparison to proto-goals

Thank you for the quick follow-up and for raising your score.

We have now run the comparison you asked for: Proto-goal RL on state-based Taxi, with the same state features and reward information as our method; see Fig 15 of the revised PDF. Proto-goal RL needs a space of candidate goals to prune, and that space is defined by its arity: a 1-hot proto-goal fixes one state feature to a value, a 2-hot proto-goal fixes two, and so on. With 1-hot proto-goals (∼30\sim 30 candidates) it matches our agent almost exactly: both exceed 0.95 success by ∼4\sim 4M frames and both end up with about 30 options. With 2-hot proto-goals (∼469\sim 469 candidates) it is far slower, reaching ∼0.87\sim 0.87 by 20M frames, below every other agent.

This demonstrates that the number of proto-goals grows combinatorially with arity and with the number of state dimensions; pruning and sampling allow proto-goals to keep making progress, but it still starts from that combinatorial set. Abstract Subgoals does not enumerate: for each novelty event, attribution proposes the one feature combination that explains it, whatever its arity, so the number of subgoals the agent learns about is the number of interesting events it encounters, not the number of possible combinations. In Taxi, single-feature subgoals suffice, so the 1-hot space happens to be the right one and our agent recovers the same subgoal set without being told the arity. When the problem requires multi-feature subgoals, our agent finds them automatically: many of the Montezuma’s Revenge subgoals in Figs. 3 and 9 attend to three features (e.g., the player, the sword and a skull), as do the relational subgoals in MiniGrid (Fig. 14: the agent’s position together with the key’s). Enumerating 3-hot proto-goals would mean pruning a space that grows as the cube of the number of candidate regions, and the 2-hot result above already shows what that costs in terms of sample efficiency in a simple domain like Taxi.

Reviewer_GDRMOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper addresses exploration in reinforcement learning by discovering abstract options that provide starting points for further exploration. The proposed method, Abstract Subgoal Option Discovery, combines intrinsic reward estimation, visual feature extraction, and feature attribution to identify relevant features and construct subgoal classifiers. It learns goal-conditioned option policies alongside an exploration policy driven by intrinsic and extrinsic rewards. Exploration proceeds by reaching promising subgoals and exploring from them. Experiments demonstrate improved performance on three exploration tasks with image observations.

Critical strengths
  • Motivation: The paper addresses an important problem with a clear and convincing motivation. The effectiveness of hierarchical reinforcement learning depends strongly on the chosen abstractions, and discovering useful abstract options is a promising direction.

  • Novelty: The method combines existing exploration techniques in an interesting way. In particular, using feature attribution to identify the features responsible for novelty peaks provides a concrete mechanism for constructing abstract subgoals.

Critical weaknesses
  • Method complexity: The method combines numerous interacting components for feature extraction, novelty attribution, subgoal classification, and option selection. Although the appendix includes useful ablations, the empirical analysis does not sufficiently establish which design choices are necessary or whether simpler alternatives could achieve comparable performance. The dependence on external feature extraction also raises questions about applicability beyond the evaluated domains.

  • Limited empirical evidence: The quantitative evaluation is narrow relative to the complexity of the proposed method. Montezuma’s Revenge is a challenging exploration benchmark, but the other two environments are structurally simpler. Moreover, the hierarchical baseline relies on near-exact full-image matching, limiting the conclusions that can be drawn about advantages over more flexible state-reaching approaches. The qualitative examples illustrate classifier generalization but provide limited evidence that the resulting option policies transfer effectively across contexts.

  • Clarity of exposition: Although the individual components are described and formalized, the overall method is difficult to follow because its specification is spread across the background, method, and implementation sections. A more concise, unified presentation would make the interactions between components easier to assess.

Questions for authors
  • Can you evaluate the method on additional challenging exploration benchmarks, such as other sparse-reward Atari games, to assess its broader applicability?

  • Can you provide ablations that remove or simplify individual components to establish which are essential to performance and justify the method’s complexity?

  • Can you compare against stronger goal-reaching baselines, for example using learned contrastive goal representations rather than near-exact pixel matching?

5 comments

We thank the reviewer for their constructive feedback. We are glad that they found the paper to be well motivated and novel. We believe that we have addressed most of their concerns through new experiments, and welcome more discussion.

Weaknesses

Method complexity. Each of the four added components (feature extraction, attribution, directed exploration, policy over options) now has an ablation (see response to Q2). Two are essential: attribution and directed exploration; the novelty estimator, CFN, is swappable with RND, the agent is fairly robust to changes in thresholds, and the segmentation is not needed when the domain has factored state inputs. On external feature extraction: in Appendix G, we evaluated our method on state-based Taxi and Ant U-Maze, precluding the need for SAM or contour detection. When domains have image-based observations, we still assume access to image segmentation, which the paper has always been upfront about.

Limited empirical evidence. We added Ant U-Maze, a continuous-control exploration task with no hand-given goal space, where our agent recovers the xy goal space that prior methods hard-code and matches an oracle-goal-space Deep Skill Graphs (DSG) agent (response to Q1). On the baseline, we added a StateSimilarity baseline which--unlike State Equality--uses a per-domain tuned threshold and is now in all three domains (response to Q3). On transfer, we added two direct tests of option policies, not just classifiers: an option discovered in room 1 of Montezuma’s Revenge reaches its subgoal from rooms 2 and 6 (Appendix E), and in Ant U-Maze one option reaches its subgoal from four distant start cells in 399/400 attempts while composed options reach held out targets at 0.9–1.0 success rate (Appendix G.2).

Clarity of exposition. We will keep trying to simplify and improve the writing of the paper.

Questions to answer

  1. Can you evaluate the method on additional challenging exploration benchmarks?

We added two domains, chosen to test applicability along two axes our three image-based domains do not cover: continuous actions, and state-based observations that need no image segmentation.

  • Ant U-Maze (Appendix G.2; sparse reward). A standard HRL testbed, but prior methods hard-code the goal space as the ant’s xy position (e.g., HIRO, DIAYN, DSG). Our agent discovers on its own that xy is the relevant subspace (Fig. 16: subgoals in the first ~2M steps are dominated by x, y dimensions) and reaches the same final success rate (≈0.8) as DSG given the oracle xy goal space, at ~3× the samples; flat TD3+CFN never solves the task. The discovered options are reusable: a single posture-seeking option reaches its subgoal from four distant starting locations in 399/400 attempts (Fig. 15), and composing options reaches target locations never used in training at 0.9–1.0 reach rate, where conditioning the same policy directly on the target drops to 0 (Fig. 18).
  • State-based Taxi (Appendix G.1; sparse reward). The same algorithm with the image segmentation step deleted. Our agent exceeds 0.95 success by 4M frames, versus ~10M for CFN and ~17M for R2D2 and State Equality. Within the rebuttal window we prioritized these two domains because they answer the question several reviewers raised–whether the method depends on the image pipeline–which another Atari game would not.
  1. Can you provide ablations that remove or simplify individual components to establish which are essential to performance and justify the method’s complexity?

Beyond a standard goal-conditioned RL agent, the method adds four components: candidate feature extraction, feature attribution, novelty-driven exploration, and a policy over options. New ablations in the revision:

  • Directed exploration (D.4). When exploring from the frontier, we replaced the CFN exploration policy with a uniform random exploration policy, keeping everything else the same. The resulting agent did not get any reward in Montezuma's Revenge.
  • Feature extraction (Appendix G). We removed the segmentation step and successfully solved state-based domains (Taxi, Ant U-Maze) with our method.
  • Policy over options (D.5). During goal selection, we select goals randomly instead of scoring goals by their extrinsic and exploration value, keeping everything else the same. The resulting agent heavily underperforms in Montezuma’s Revenge.

This is in addition to 3 pre-existing ablations:

  • Feature attribution (Sections 4.4, D.2, G.1). Change only the classifier from the selected features to the full state. Repeatedly, the evidence suggests that our feature attribution is key to good performance.
  • Novelty estimator (D.3). Swap CFN for RND. Similar features selected, albeit less cleanly because RND’s Shapley values are more evenly distributed across candidate features.
  • Threshold hyperparameter sensitivity (D.1). Sweep both multipliers over {1, 2, 3}σ; most settings reach 1,000+.
  1. Can you compare against stronger goal-reaching baselines, for example using learned contrastive goal representations rather than near-exact pixel matching?

    Based on your suggestion, we added two stronger baselines:

  • State Similarity (Section 4.4, Fig. 5, all three domains): the full-image distance test with the threshold tuned per domain for best final return (Table 5), i.e., the most favorable version of state-reaching with this classifier. It improves on State Equality in every domain and matches our method in VisualTaxi, but remains well below it in MiniGrid-KeyCorridor and Montezuma’s Revenge.
  • Deep Skill Graphs with an oracle goal space (Appendix G.2): a hierarchical agent handed the xy goal space, which is what a contrastive or other learned goal representation could at best recover in this domain. Our agent, which must discover the goal space, reaches the same final success rate.

I appreciate the authors efforts to address my concerns. However, I am still not convinced of the method’s generality for discovering abstract options. My main concern remains the fixed feature extractor, which is generally not a realistic assumption. The new AntMaze experiment also does not show general applicability, since the single-state features are given to the method rather than learned. I still believe abstract option discovery is a worthwhile goal, but the method remains too convoluted for its performance to justify it. I will therefore keep my score, and I wish the authors good luck in further developing the paper.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.