- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes a new benchmark for continual reinforcement learning called ArcadeWorld, which aims to capture three important properties of continual learning: continuous improvement, non-ergodicity, and a world larger than the agent. The authors introduce leaderboard rewards, where the agent's past performance influences future rewards, and provide a formal characterization of their properties. They develop two environments, BitWorld and AtariWorld, to study this setting. Their experiments show that standard deep RL algorithms struggle in these environments, often performing close to a random policy and below simple memory-based or intrinsic-exploration baselines.
Critical strengths
-
A useful framework for continual RL with theoretical foundations. The paper proposes an interesting framework that adapts existing episodic environments into continuing learning problems, where the agent's previous performance affects future rewards. The formal characterization of leaderboard rewards provides theoretical support for the proposed formulation. I believe this could be a useful benchmark for the continual RL community.
-
Interesting empirical findings that motivate further research. The experiments on BitWorld and AtariWorld show that several established deep RL algorithms struggle in these settings, sometimes performing worse than simple memory-based or intrinsic-exploration baselines. These results highlight interesting challenges and motivate the development of algorithms that can handle continuing, history-dependent learning problems.
Critical weaknesses
The causes of the observed failures are not sufficiently isolated. While the experiments show that standard deep RL algorithms struggle in BitWorld and AtariWorld, it is unclear which aspects of the proposed setting are primarily responsible. The paper emphasizes exploration as a key challenge, but other factors, such as sparse rewards, credit assignment, optimization difficulties may also contribute. More controlled experiments would help distinguish these explanations and strengthen the conclusions about the limitations of existing algorithms. It is also unclear how sensitive the results are to the choices of the hyperparameters or whether stronger exploration-oriented methods could substantially improve performance. These limitations make the broader conclusions about the difficulty of the proposed framework less convincing.
Questions for authors
Q1. The paper identifies exploration as a key challenge in the proposed environments. How can the authors distinguish exploration difficulties from other possible causes of failure, such as credit assignment, reward sparsity, or optimization instability? Are there additional experimental results that support this interpretation?
Q2. How sensitive are the experimental results to hyperparameter choices and algorithm adaptations? In particular, have the authors considered stronger exploration-oriented baselines or additional tuning for the continuing setting?