- Rating
- 2
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
This paper studies the problem of efficient on-policy reinforcement learning. To address this issue, the paper presents a new method, in which expert and learner alternate control in fixed-length blocks, with a learnable probability determining expert participation. An expert-use penalty encourages withdrawal, while imitation on expert-controlled blocks trains the learner from unused experience. Experiments across 25 DMControl and nine MiniGrid tasks show improved sample efficiency and aggregated final performance.
Critical strengths
- The paper provides a simple interpretable mechanism for adapting expert participation.
- The experimental results are sufficient and evaluate the learner independently of the expert.
Critical weaknesses
- The paper reports the improved performance in terms of normalized return. However, it seems the improvement is marginal.
- The suboptimal policy from the expert affects the final policy generated by both the expert and the learner when adopting blocks. It may result in long training time and high computational costs.
- It could discuss how to balance the block size.
- Eqs. (4) – (13) use complete block rewards, boundary-state values, and scores accumulated over an entire control block. However, Table 5 specifies a PPO rollout length of 30 steps and a control-block length of 100 steps. Thus, a complete block extends across multiple PPO collection windows. The paper does not explain this and its implications.
Questions for authors
- How to define the price of expert use? Please clarify this.
- How is the expert share automatically adjusted during the process?
- It is not clear how to form a block. How many steps are formed as a block? How the block size affects the performance? It would be better to add an experiment to assess this.
- It is unclear whether policy parameters remain fixed across PPO collection windows. Please explain it.