acceptodds
Under review as a conference paper at ICLR 2027

Discovery of Sticky and Responsible Options for Frozen-Option Transfer

Abstract

A fundamental problem in hierarchical reinforcement learning is finding a small number of reusable and easily composable behaviors that transfer with little or no fine-tuning to solve novel tasks. Option and skill discovery frameworks are two prominent approaches that address this problem. However, option discovery typically focuses on learning specialized behaviors for a training task, leading to a low diversity of behaviors with limited transferability, and unsupervised skill discovery learns diverse behaviors that can be misaligned with downstream tasks. To address these issues, we introduce Task-Agnostic Sticky and Responsible Options (TASRO), an option-learning framework that combines (1) Markovian option switching at every time step, (2) an objective promoting persistent, diverse, and specialized options, and (3) information asymmetry, where the high-level controller observes both task and task-agnostic information while the low-level options observe only task-agnostic information. We evaluate under a strict frozen-option transfer protocol: pretrain options on a goal-reaching training task, freeze them, and relearn only the controller on downstream tasks. Across our navigation, locomotion, and manipulation benchmarks, TASRO improves frozen-option transfer over the tested skill-discovery and task-exposed option baselines.

Primary area:
reinforcement learning
Keywords:
hierarchical reinforcement learning, option discovery, skill discovery, frozen-option transfer, temporal abstraction, information asymmetry, transfer learning, continuous control

Reviews

Among:
Shape:
1234this paper, Bottom 25%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.75 over 4 reviews, above 25% of the others.
Reviewer_8mqVOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes Task-Agnostic Sticky and Responsible Options (TASRO), a hierarchical reinforcement learning framework designed to address the task specialization problem of reward-driven option discovery and the downstream misalignment of unsupervised skill discovery. The proposed method combines Markovian option switching, sticky and responsible options, and information asymmetry. For downstream tasks, the learned low-level options are frozen while a new high-level controller is trained, which is called frozen-option transfer. Experiments across navigation, locomotion, and manipulation benchmarks show improved frozen-option transfer performance compared to option-learning and unsupervised skill discovery baselines.

Critical strengths
  1. The paper addresses an important challenge of learning a compact set of reusable options from a single training task, without requiring expert demonstrations or multi-task training.
  2. The proposed framework is carefully designed, to provide a coherent regularized objective with analytically derived option and controller updates. Also, conditioning both option selection and execution on the previous option provides additional flexibility compared to classical call-and-return options. The overall framework is technically sound.
  3. The paper demonstrates impressive performance across broad benchmarks, outperforming baselines mostly under frozen-option transfer setting.
Critical weaknesses
  1. In the frozen-option transfer experiments, TASRO outperforms the baselines in HumanoidMaze and Panda manipulation but underperforms PPOC* in AntMaze Hardest and Barkour Predator. These results raise concerns about the framework's robustness, particularly when downstream tasks require behaviors that are insufficiently represented in the source training task. As the authors acknowledge, successful transfer depends heavily on selecting a suitable training task that exposes behaviors useful for downstream tasks. This dependency limits the general applicability of the proposed framework.
  2. Although the paper introduces various components intended to improve transferability, the ablation studies in Section 3.3 only report results on training tasks. Consequently, the individual contributions of Markovian option switching, stickiness, responsibility regularization, and information asymmetry to downstream transfer remain unclear.
Questions for authors
  1. Could the authors further investigate how sensitive TASRO's transfer performance is to the choice of a training task? Are there practical strategies to improve transferability without requiring a carefully selected source task that already exposes behaviors useful for downstream tasks?
  2. Could the authors provide downstream transfer ablations that independently remove Markovian option switching, the sticky prior, responsibility regularization, and information asymmetry, while keeping the underlying training algorithm and computational budget fixed?
  3. Since TASRO conditions the option policies on the previously active option, the same option can generate different behaviors depending on the preceding option. Could the authors provide an ablation study comparing the proposed option design with one that does not condition on the previous option, to isolate the performance gains attributable to this additional representational capacity?
Reviewer_UAeJOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

This paper introduces Task-Agnostic Sticky and Responsible Options (TASRO), a hierarchical reinforcement learning method for discovering reusable low-level behaviors.

The method invovles a high-level controller that receives task information, and an options framework, where the options receive a restricted observation that excludes task-specific variables. The controller selects options at every training step and conditions its decisions on the previously active option.

The method includes additional regularization objectives to encourage the options to be persistent, diverse, and specialized for different scenarios. After training on a single goal-conditioned task, the options are frozen, and a new controller is trained for each downstream task.

The paper evaluates this approach on AntMaze, HumanoidMaze, Barkour, and Panda manipulation, all in simulation. The key question the authors attempt to address through experiments is whether the learned behaviors remain useful when recomposed without low-level fine-tuning.

Critical strengths

The paper addresses an important problem in reinforcement learning: learning behaviors that can be reused rather than requiring re-training for every task. I like that the evaluation keeps the learned options frozen on downstream tasks. This is helpful in testing whether the behaviors can be reused as they are, instead of relying on fine-tuning to adapt them to each new task.

The design of the merthod is well motivated at a high-level. Restricting task information at the low level aims to reduce task-specific behavior. Conditioning on the previous option also allows the approach to account for transitions between behaviors, instead of treating each option independently.

The evaluation includes several baselines, multiple training seeds, and analyses of individual option behavior. The HumanoidMaze results are encouraging. The experiments also provide insights into the method beyond comparison with baselines, and attempt to illustrate what the learned options actually do.

Critical weaknesses

The evaluation does not cover sufficiently varied and demanding transfer settings. Although evaluations are on four domains, two of them focus on maze navigation. The Barkour tasks remove terrain, obstacles, and external disturbances, and the manipulation tasks include only one pick-to-push transfer. More complex locomotion and multi-stage manipulation is necessary to better understand the proposed benefits of the approach.

The main limitation of the paper is that there are no real-robot results. All experiments are conducted in simulation, leaving the practical value of the frozen options unverified. Real-robot exepriments are essential to understand the practical benefits of the approach.

The transfer results do not show a consistent advantage. TASRO performs substantially worse than PPOC* on Barkour. On Panda, it improves over the frozen-library baselines but performance remains lower than PPO trained from scratch. Although these methods follow different learning protocols, the results leave the practical benefit of frozen reuse unclear.

The paper is difficult to follow, and the main technical contributions are not clearly and crisply articulated. The proposed novelty lies in combining several design choices. However, the main text presents a three-term objective, while the experiments use additional entropy corrections introduced in the appendix. A clearer description of the complete method and its components is necessary.

I am less convinced by the evidence for the additional mechanisms. The detailed transfer ablations focus mainly on AntMaze, where information masking has a large effect, while stickiness and previous-option conditioning provide smaller and unclear gains. These results leave it unclear whether this additional complexity in the method consistently improves transfer across domains.

Questions for authors

How well does TASRO transfer to more varied and challenging downstream tasks? I would like to see evaluations on obstacle-rich locomotion and multi-stage manipulation to understand whether the learned options remain useful when task requirements change more substantially.

Could you include real-robot experiments to test the method beyond simulation? In particular, it would be useful to understand how real-world organic variations affect both the individual behaviors and the transitions between options.

Could you explain and provide additional context on why TASRO transfers less effectively on Barkour and performs worse than PPO on the Panda env in simulation?

Please clarify what is new in TASRO relative to the closest option-learning methods? It would also help to write out the full objective used in the experiments and explain how the additional entropy corrections relate to the main formulation.

Reviewer_9fkgOpenReview ↗
Rating
3
Confidence
1
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

The paper presents a new approach to hierarchical RL based on "sticky and responsible" options. As far as I understand it, this means a prior favoring stickiness, for more consistent and persistent action, and an additional loss favoring sharper specialization of options so their responsibility for downstream consequences in RL can be more appropriately and specificailly attributed.

Critical strengths

The paper is clear and interesting. I am not an expert and really cannot assess its strengths but I was happy to read it.

This is also the only paper I was assigned where the authors clearly did not use AI to drive most of the research -- this is clear from both the writing, and the authors' use of AI statement. I applaud them for this. Although I am very far from this paper's area of focus and I have little to no expertise in hierarchical RL, relative to all the other, mostly-AI-generated papers I had to review, this one struck me as the most valuable research contribution and was certainly the most clearly written.

Critical weaknesses

None that I can tell, but I am not an expert in this field.

Questions for authors

No questions, but mostly because the paper was clear and I don't feel I have sufficient expertise in this topic to probe more deeply.

Reviewer_b6RkOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
3
Read review
Paper summary

This paper studies discovery of temporal abstractions in RL. Their method attempts to find options/skills (I will use those terms interchangeably) that are temporally extended, and also diverse, meaning that different options should result in different behaviors. Their method has 3 technical components: (1) per timestep switching, (2) diversity and specialization balancing, and (3) information asymmetry. By the last one, they mean that the intra-option policy and the policy over options both get different state variables that are decided by the designer. They compare their method, TASRO, against flat and hierarchical baselines in MuJoCo type domains.

Critical strengths
  1. The frozen option test is a good one to isolate transfer.
  2. I found equation 3 very interesting. It brings together ideas from many different literatures.
  3. Experiments were thorough from what I could tell, and they have good wins for TASRO in HumanoidMaze and Panda; I know the former is a hard problem.
Critical weaknesses
  1. The intro's claims are not supported by the experimental results. PPOC* is an option-learning baseline that wins on Barkour, and TASRO loses to flat PPO on Panda.
  2. Big gains in performance come from the information asymmetry. This is a well known result in HRL and the challenge in the field is to discover the two sets of variables automatically from interaction data, which this paper does not tackle.
  3. The per-timestep switching is very similar to the generalized policy improvement literature of Barreto et al (e.g., option keyboard, geometric policy composition (thakoor et al, 2022)) and I couldnt understand what is new in this paper versus established in that literature which is continuing to develop now. The paper should engage with that literature as well since it claims novelty in that regard.
  4. I liked the comparison to PPOC* but from what I could tell, TASRO uses SAC as its base optiimizer, right? Why not compare against a PPOC* style method but with SAC in the mix so that we are not comparing on-policy (apples) to off-policy (oranges)?
  5. The sticky prior seems very similar to Harb et al 2018 work on deliberation costs in option critic, which has many follow on papers. I dont see the novelty of their sticky prior compared to that line of work.
Questions for authors

See above.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.