- Rating
- 1
- Confidence
- 2
- Informativeness
- 1
- Presentation
- 1
- Soundness
- 1
Read review
Paper summary
Within the option-critic framework, the paper studies the phenomenon called policy necrosis, where intra-option policies stop exploring and commit too early to an arbitrarily sampled primitive action. The phenomenon arises when the step size is large and the intra-option policy is a softmax policy with a temperature parameter that is too low. The paper identifies -greedy as a solution to this problem.
Critical strengths
None
Critical weaknesses
- Writing/Clarity/Structure
- The paper shows signs of heavy use of AI for writing. As a result, there exists sentences or phrases that are unprofessional, vague, or simply incorrect. To name a few just from the beginning of the paper:
- In the abstract: "the termination rule option-critic learns by maximizing return contributes nothing." This sentence is simply vague, and very likely overstated.
- The abstract also claims that the "headline result" of option-critic is that "performance improves as options are added." I believe this statement is simply incorrect. The main contribution of the option-critic framework is end-to-end discovery of options. The prior work is unfortunately misrepresented.
- Last paragraph of the introduction: "The three results compose." Unfortunately I'm not sure what it means for results to "compose".
- What is in Theorem 2? I don't think i saw a definition.
- The paper shows signs of heavy use of AI for writing. As a result, there exists sentences or phrases that are unprofessional, vague, or simply incorrect. To name a few just from the beginning of the paper:
- Limited scope
- The paper focuses only on four rooms for the experiments and focuses on one particular implementation of option critic with specific hyperparameter values. It is unclear whether the findings generalize to other environments/settings.
Both the scope and the level of writing in this paper falls far below what I would expect from an ICLR paper.
Questions for authors
The authors noticed that when the step size is large and the temperature for the softmax policy is low, option policies could commit too early to a random action. Before studying this phenomenon, running experiments and proposing solutions, have the authors tried hyperparameter tuning?