- Rating
- 4
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
The paper analyzes the impact of spatially and temporally sparse rewards on policy learning for a particular type of problem, i.e., pursuing a moving target in a continuous space.
Critical strengths
- The direction is interesting and relevant to the RL community. It provides structural insights into the impact of spatial and temporal factors of reward sparsity which could help to devise better algorithms.
- The paper is well-motivated and presented with a clear outline, notation, and illustrations. Since I am no expert in this field, there is a chance that I missed some details, though.
Critical weaknesses
The paper focuses on a very specific problem type (i.e., moving target pursuit), which limits the generality of the analysis.
Questions for authors
None
Rebuttal
Thank you for reading our paper and for your kind words. We did our best to make some highly technical content comprehensible, and are very glad you found it clear.
We absolutely agree that our focus was on a very specific problem type. We think this focus was reasonable given that considering a specific setting allows us to perform a much more mathematically detailed analysis, but it does leave open questions related to how well our conclusions generalize to more realistic settings (e.g., nontrivial environmental transitions, value learning, sampling noise...). Although a broad empirical study would probably bloat the paper too much, we still think it could be interesting to include some empirical results for more realistic environments. We've done some pilot studies like this already, and so far our main conclusions (e.g., spatial sparsity can cause exponential slowdowns) have held even if various assumptions are relaxed.