- Rating
- 3
- Confidence
- 3
- Informativeness
- 3
- Presentation
- 2
- Soundness
- 3
Read review
Paper summary
The authors provide a construction to demonstrate that discounted Monte Carlo optimistic policy iteration under nonuniform state sampling can cycle. Then, they prove that the original stochastic recursion also does not convergence with high probability under suitable initialization and stepsize conditions.
Critical strengths
- I think the problem is interesting, it's a problem that's been "open" for over 20 years. Although its practical significance may be debated, I think resolving this longstanding question is a worthwhile theoretical contribution.
Critical weaknesses
- I think there are issues with the presentation, to the point that it interferes with comprehension at places. For instance, at lines 60-61, it's not immediately clear that denotes an action (I admit it is inferable given some thought, but still). Again, in equation (3) (lines 70-71), is used without being previously defined, and it again falls to the reader to interpret what it is. Introducing notation before or alongside its first use would make the setup easier to follow. I would encourage a careful pass through the manuscript to address such issues. A minor comment is that the AI-isms are sometimes distractingly heavy (things of the sort "this is X, not Y" etc.)
Questions for authors
- What does this result tell us, in that, how does it affect the way we would use Monte Carlo OPI in practice? Given the initialization and stepsize conditions in the nonconvergence theorem, how should practitioners judge the risk of cycling in their own applications?