- Rating
- 1
- Confidence
- 4
- Informativeness
- 1
- Presentation
- 1
- Soundness
- 1
Read review
Paper summary
The paper proposes a new method to improve exploration and sample efficiency in sparse-reward environments. The main idea is to incorporate a notion of task progress or relevance into the reward function and replay buffer priorities, encouraging the agent to visit states that contribute to reaching the task goal.
Critical strengths
Please refer to the weaknesses section.
Critical weaknesses
-
My main concern is the clarity of the paper. There is no clear explanation of how the progress signal is determined in each environment. Is this signal manually defined for each environment? If so, this raises concerns about the generalization of the proposed method, as there is no clear procedure for defining in new environments.
-
The literature review is very limited, and the paper could have discussed more existing works on improving exploration in sparse-reward settings.
-
I had a difficult time understanding the paper and could not clearly identify the motivation behind the proposed method. For instance, what is Figure 2 actually illustrating?
-
Based on Figure 3, the proposed method outperforms the baselines in only 3 out of 12 environments. In the remaining environments, all methods achieve approximately the same final success rates.
-
Another concern is the motivation behind the proposed method. In environments with distractors, the policy should learn to avoid those distractors and continue toward the goal. However, the proposed method appears to avoid the states where distractors are located rather than learning how to act appropriately in those states. As a result, if the policy is initialized in one of these states, it may not be able to select an appropriate action.
Questions for authors
Please refer to the Weaknesses section.