- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
the paper tries to understand when deep RL performance is poor, whether that is caused by lack of good experience due to exploration issues, or lack of ability to exploit good experience. prior work measures optimality gap relative to an oracle policy or human policy, which conflates the two, so this work introduces the notion of experience-best policy. they show a significant gap pointing to poor exploitation on both deterministic and stochastic environments. they also show the gap increases with RND and with scale.
Critical strengths
I think the idea of optimality gap relative to experience is nice and does untangle whether the issue is poor experience or poor exploitation. If measured appropriately it could be a valuable diagnostic for practitioners and the research community. However see weakness section for discussion on rewarding good outcomes vs decisions.
Critical weaknesses
I believe the paper does not adequately address the crucial issue of disentangling luck from better policy in computing the experience optimal policy. Do we reward good outcomes or good decisions? I think we should reward good decisions here but it seems like the proposal is rewarding good outcomes which could be entirely luck in the worst case. They simply define the experience optimal policy return by replaying the top k% best trajectories (possibly recent). One can imagine a simple MDP where taking action A deterministically transitions to terminal state and gives reward 1, while taking action B is a lottery between reward 10 with probability (k+eps)% and reward -inf with remaining probability. Here the experience optimal policy would collect reward 10 by luck. The optimal policy would take action A and collect reward 1.
Also it seems like we need to track 2 numbers to achieve the paper's stated aim, not just the gap between best experience and learned policy, but also the value achieved by the learned policy (or best experience). Just looking at the gap is uninformative, which the authors point out and say they also check to make sure the policy learns, but I think a binary check is insufficient and we need both numbers to compare.
Questions for authors
- in figure 4 are you tracking the raw gap between experience optimal and learned policy? what about the normalized gap relative to experience optimal? shouldn't the ratio be what matters, not the raw value?
- how do you address the luck issue?
- could something like the following help disentangle luck vs good decisions? resample the highest value trajectories (replay those actions where possible until things diverge) and if there's regression to the mean that points to luck?