- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes Task-Agnostic Sticky and Responsible Options (TASRO), a hierarchical reinforcement learning framework designed to address the task specialization problem of reward-driven option discovery and the downstream misalignment of unsupervised skill discovery. The proposed method combines Markovian option switching, sticky and responsible options, and information asymmetry. For downstream tasks, the learned low-level options are frozen while a new high-level controller is trained, which is called frozen-option transfer. Experiments across navigation, locomotion, and manipulation benchmarks show improved frozen-option transfer performance compared to option-learning and unsupervised skill discovery baselines.
Critical strengths
- The paper addresses an important challenge of learning a compact set of reusable options from a single training task, without requiring expert demonstrations or multi-task training.
- The proposed framework is carefully designed, to provide a coherent regularized objective with analytically derived option and controller updates. Also, conditioning both option selection and execution on the previous option provides additional flexibility compared to classical call-and-return options. The overall framework is technically sound.
- The paper demonstrates impressive performance across broad benchmarks, outperforming baselines mostly under frozen-option transfer setting.
Critical weaknesses
- In the frozen-option transfer experiments, TASRO outperforms the baselines in HumanoidMaze and Panda manipulation but underperforms PPOC* in AntMaze Hardest and Barkour Predator. These results raise concerns about the framework's robustness, particularly when downstream tasks require behaviors that are insufficiently represented in the source training task. As the authors acknowledge, successful transfer depends heavily on selecting a suitable training task that exposes behaviors useful for downstream tasks. This dependency limits the general applicability of the proposed framework.
- Although the paper introduces various components intended to improve transferability, the ablation studies in Section 3.3 only report results on training tasks. Consequently, the individual contributions of Markovian option switching, stickiness, responsibility regularization, and information asymmetry to downstream transfer remain unclear.
Questions for authors
- Could the authors further investigate how sensitive TASRO's transfer performance is to the choice of a training task? Are there practical strategies to improve transferability without requiring a carefully selected source task that already exposes behaviors useful for downstream tasks?
- Could the authors provide downstream transfer ablations that independently remove Markovian option switching, the sticky prior, responsibility regularization, and information asymmetry, while keeping the underlying training algorithm and computational budget fixed?
- Since TASRO conditions the option policies on the previously active option, the same option can generate different behaviors depending on the preceding option. Could the authors provide an ablation study comparing the proposed option design with one that does not condition on the previous option, to isolate the performance gains attributable to this additional representational capacity?