- Rating
- 1
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper presents an additional module that enables skill discovery methods to learn skills that return to the initial state at the end of an episode. The proposed method applies to a wide range of SD methods. The methodology mainly involves adding phase information to the policy's input to ensure that the policy is aware of the current phase, and flipping the reward sign after the midpoint of the trajectory to reward returning behavior. Other techniques like return constraints are used to ensure the learning of returning skills. Experiments are conducted on a wide range of domains and base algorithms, demonstrating that RSD can learn returning skills and faithfully showing the advantages and weaknesses of the learned skills.
Critical strengths
- The problem is new and interesting.
- The proposed method is easy to understand and applies to both DSD methods and MI-based SD methods.
Critical weaknesses
- The proposed method introduces many hyperparameters and seems to be sensitive to hyperparameter tuning, which makes it hard to use in practice. As shown in Table 3, every environment and base algorithm requires different RSD parameters.
- Page 3 lines 147-150 say that "RSD leaves the base objective untouched, and what it changes is determined by whether that objective’s reward admits a potential in the sense of Equation (3)". However, on page 5 line 268-269, the authors states that they additionally phase-conditions the dynamics model of DADS. This suggests that RSD also requires individual changes for certain algorithms, which makes it not a method-agnostic add-on as claimed in the abstract.
Questions for authors
- How will RSD perform if using the same hyperparameters across all tasks and base algorithms? How will the performance change if these hyperparameters change?
- How will RSD perform if we flip the reward at other timesteps, like 1/4 or 3/4?
- The proposed algorithm does not seem to be applicable in scenarios that contain non-reversible parts. The forward and return dynamics may be asymmetric in practice. For example, it is easy to throw away an object held in the arm, but picking it up is more challenging. Such scenarios are revertable but hard to revert. How will RSD perform in such scenarios?
- Page 5, line 217, How is the O(1/T) derived?
- Why do we need returning skills if combining several non-returning skills can achieve the same result? A larger K, as stated in 5.2, is one reason. But a too-large K will degrade the hierarchical control into a process of first selecting then finetuning a certain skill, which is no longer hierarchical. How will RSD perform if using the same K as is used in the hierarchical control experiment in the original paper of the base SD algorithms?
- In Appendix A.2, prop.3, the authors make some assumptions on |Δft| and Δft*. However, these assumptions are not used in the proof. What are these assumptions for?
- (minor) Notation issues: e(tau) is defined in (5) but is not used. Kw in (7) is not defined. ξ in (9) is not defined (I see ξ∼π, but please explicitly define ξ with states and actions).