- Rating
- 2
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
This paper presents an approach for learning options and option policies for sparse reward RL problems. The core idea is to use novelty in observed state sequences as a surrogate for detecting subgoal candidates, and to then use image-based feature extractors to identify parts of images that contribute the most to their corresponding states' novelty. Shapley value based credit assignment is then used to identify relevant features and subgoals are created for achieving salient feature values. The central claim of the paper is that unlike related work, this work defines subgoals based on features rather than full states.
While the ideas presented are interesting and result in feature-based options that are potentially more decomposable and reusable than many other approaches for learning options, some of the claims of novelty are difficult to support.
Critical strengths
-
Discovery of salient features and feature based subgoals has the potential to support greater transfer across problem instances.
-
Good empirical analysis on three test domains
-
Paper is well written, with ideas that are clearly motivated
Critical weaknesses
-
Transfer performance is not evaluated quantiatively
-
Claims that the presented approach transfers seamlessly to factored state representations are not supported by theoretical or empirical evidence. In particular it is not clear that the presented approaches for feature discovery would work equally well in factored domains, and recent approaches in that direction (e.g. Nayyar et al., AAAI 2025) don't use each dimension as a candidate feature. Empirical comparison with such approaches would help justify these claims.
-
It is difficult to argue that this is the first work that goes beyond treating full states as subgoals. Prior work has considered more general notions of subgoals. E.g., Bagaria et al. (IJCAI 2021) define option termination conditions using classifiers; other examples include Veeriah et al., NeurIPS 2021. Much of the work on learning subgoals in latent spaces implicitly achieves the desired concept of defining subgoals in terms of sets of states rather than single states; the difference here could be argued to be not merely going beyond state-reaching, but in terms of learning decomposable subgoal features that can be recomposed in novel ways for novel reward functions. Such a claim would be more justifiable.
Bagaria et al. "Robustly learning composable options in deep reinforcement learning." IJCAI 2021.
Veeriah et al. "Discovery of options via meta-learned subgoals. NeurIPS 2021.
Nayyar et al. "Autonomous option invention for continual hierarchical reinforcement learning and planning." AAAI 2025.
Questions for authors
-
Are the interactions for learning subgoals, intra-option policies and inter-option policies included in Fig. 5?
-
The phases mentioned in the previous question are cleanly separated as different modules in the overall algorithm. Can you provide a breakdown of the number of interaction steps required for the components listed above?
-
Have you tried to implement and evaluate this approach for transfer across problem instances? It seems that transfer performance would be a better metric for distinguishing your approach from existing work rather than merely learning options that go beyond state-reaching.
-
Can you provide data justifying the additional cost of computation required for feature discovery, pruning, and learning inter and intra-option policies vs flat RL? E.g., clock-time based learning curves including including all the steps required for the presented approach.
5 comments
Today is the last day of the discussion period, so we wanted to check in to see if there’s anything else you would like addressed; we believe we have cleared up all your concerns.
Thank you for responding to the concerns I raised in my review. I think the shift towards evaluating generalization and transfer is the right approach, however, it requires substantial new analysis and a clear, well formulated empirical evaluation. In particular, it appears that this approach applies more to navigation tasks than to more general settings. For instance, in Montezuma's revenge the learned subgoal highlighted in Appendix E highlights a generic subgoal (reach "Center left" of screen). Such goals are in fact too non-discriminative: they would increase the effective branching factor since it is not clear when they will be useful and it is not clear how they could be eliminated. In addition, the extra GPU/computational cost needs to be considered carefully and reported with the experiments for all domains. Finally, there are several approaches for discovering options in static as well as continual RL settings with state based input or factored representations. Pointers for these methods can be found in the papers discussed above.
I encourage the authors to continue developing this thread of research. However, I also believe that a version addressing these concerns requires more extensive changes and a full review. I will therefore maintain my current score.
We thank the reviewer for a careful review. We added transfer experiments (Appendices E and G.2), two factored-state domains with the extraction step removed (Appendix G). Responses to each point follow.
Questions
Are the interactions for learning subgoals, intra-option policies and inter-option policies included in Fig. 5? Yes, all of them. The method is fully online: subgoal discovery, intra-option policy learning and the policy over options run inside a single interaction loop (Algorithm 1), with no separate phases, so the x-axis of Fig. 5 counts every environment frame the agent experiences. The only experiments that use a separate discovery-only stage are the ablations in D.2 and D.3, where it serves to hold the subgoal set fixed across methods, and this is explicitly stated there.
Breakdown of interaction steps per component. The components are separate modules in code but not separate phases in time, so there is no per-component interaction budget; every frame is used by all of them. Attribution and classifier construction operate on stored frames and consume no interaction; intra-option policies train off-policy from all frames via hindsight relabeling; the policy over options combines predictions from CFN and the goal-conditioned value function, so it also does not consume frames of its own.
Show policy transfer across problem instances. Yes, this was a major addition in the revision:
Computational cost relative to flat RL. Measured on identical hardware, Montezuma's Revenge runs take 1.25x the wall-clock of flat R2D2 for the same number of frames, and use two GPUs rather than one: the second hosts the CFN novelty model and the image segmenter. Whether or not this additional cost is justified depends on the application, we can include a note on computational requirements for the camera-ready if that helps. We have added Appendix H to answer this question; Fig 21 compares the different methods on state-based taxi in terms of wall-clock time.
Weaknesses
The claim that the approach transfers seamlessly to factored state representations is unsupported. We ran our method on two factored-state domains with the extraction step deleted (Appendix G): in state-based Taxi, the ordering of methods matches the image-based domains and our agent reaches 0.95 success in 2.5x fewer frames than CFN (G.1). In Ant U-Maze, each of the 29 state dimensions is a candidate feature, the agent discovers that xy is the relevant subspace, and it matches final performance of an agent given the oracle xy goal space (G.2).
Comparison to Nayyar et al, 2025. We thank the reviewer for the pointer to Nayyar & Srivastava (2025), which we had missed. It is relevant and we will discuss it in Section 2. Its setting differs from ours in inputs (a given set of state variables and a simulator, versus pixels) and objective (emphasis on continual multi-task planning, versus online single-task exploration), so we do not include a direct comparison. We hope that the comparison to Deep Skill Graphs with privileged access to the goal space will be a sufficient addition.
Related work that also learns subgoal classifiers. On the cited work: the termination classifiers of Bagaria et al. (2021) are learned over a designer-chosen goal space (the agent’s xy position). The oracle-goal-space agent in Appendix G.2 of the revision is that idea instantiated in Ant U-Maze, and our agent, which must discover the goal space, matches its final success rate. Veeriah et al. (2021) take the full state as input; such methods could in principle discover factored subgoals, but end-to-end representations in RL tend to be entangled (Rodriguez-Sanchez et al., 2025; Kumar et al., 2025), and the factored subgoals in our Figs. [13, 16] show that our method finds them in practice.
We would be glad to clarify anything further during the discussion period.