- Rating
- 1
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 1
- Soundness
- 1
Read review
Paper summary
The paper shows that, with limited capacity (or representation dimensionality), a policy might get "trapped" with representations that are optimal for the policy's own visiation distribution, but discourage visiting states from which it could achieve higher cumulative reward. They show that this "trap" exists across a variety of controlled experiments, and proceed with some experiments on MiniGrid and DMControl.
Critical strengths
None.
Critical weaknesses
- The paper is extremely hard to parse. It is filled with vague terminolgy (e.g., "Cross-talk allocation").
- It is very unclear what the neglected branch in DMControl and other tasks is. The authors mention that they "rebalance replay across task phases", but don't elaborate on it.
- The entire problem seems to be just an exploration problem framed as a representation problem. More concretely, if the agent can visit high reward regions, shouldn't reward give some learning signal to improve? So the experience itself is the underlying problem?
Questions for authors
- Isn't increasing the representation dimension a straightforward fix?
- Please explain the experiments on DMControl.
- How is this problem different from exploration?