acceptodds
Under review as a conference paper at ICLR 2027

Learning in a World That Remembers: When Past Performance Shapes Future Reward

Abstract

An agent can improve its general competence by directing itself towards what it can learn but has not yet learned. Yet an agent may continue to receive reward for repeating previously learned behaviour, without any incentive to broaden its competence. In this paper, we introduce the ArcadeWorld problem, where an agent chooses both its actions within individual environments and which environments in a suite to engage with. The agent’s goal is to develop general competence by improving across the entire suite within a single lifetime. We propose leaderboard rewards to formalize this goal: the agent’s past performance is remembered, so that improvement is rewarded while previously attained performance receives diminishing rewards. To understand the challenges of learning with leaderboard rewards, we conduct experiments on a simple abstraction of ArcadeWorld where continual improvement is possible. We find that deep reinforcement learning algorithms can fail to match the performance of simple memory-bounded agents, with exploration being a key challenge. To investigate these challenges at a larger scale, we introduce AtariWorld, where an agent can play and switch among Atari games. Standard deep reinforcement learning algorithms similarly struggle in AtariWorld and are even outperformed by exploration methods that receive no external reward based on score.

Primary area:
reinforcement learning
Keywords:
reinforcement learning, continual learning, lifelong learning

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_rBPSOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes a new benchmark for continual reinforcement learning called ArcadeWorld, which aims to capture three important properties of continual learning: continuous improvement, non-ergodicity, and a world larger than the agent. The authors introduce leaderboard rewards, where the agent's past performance influences future rewards, and provide a formal characterization of their properties. They develop two environments, BitWorld and AtariWorld, to study this setting. Their experiments show that standard deep RL algorithms struggle in these environments, often performing close to a random policy and below simple memory-based or intrinsic-exploration baselines.

Critical strengths
  1. A useful framework for continual RL with theoretical foundations. The paper proposes an interesting framework that adapts existing episodic environments into continuing learning problems, where the agent's previous performance affects future rewards. The formal characterization of leaderboard rewards provides theoretical support for the proposed formulation. I believe this could be a useful benchmark for the continual RL community.

  2. Interesting empirical findings that motivate further research. The experiments on BitWorld and AtariWorld show that several established deep RL algorithms struggle in these settings, sometimes performing worse than simple memory-based or intrinsic-exploration baselines. These results highlight interesting challenges and motivate the development of algorithms that can handle continuing, history-dependent learning problems.

Critical weaknesses

The causes of the observed failures are not sufficiently isolated. While the experiments show that standard deep RL algorithms struggle in BitWorld and AtariWorld, it is unclear which aspects of the proposed setting are primarily responsible. The paper emphasizes exploration as a key challenge, but other factors, such as sparse rewards, credit assignment, optimization difficulties may also contribute. More controlled experiments would help distinguish these explanations and strengthen the conclusions about the limitations of existing algorithms. It is also unclear how sensitive the results are to the choices of the hyperparameters or whether stronger exploration-oriented methods could substantially improve performance. These limitations make the broader conclusions about the difficulty of the proposed framework less convincing.

Questions for authors

Q1. The paper identifies exploration as a key challenge in the proposed environments. How can the authors distinguish exploration difficulties from other possible causes of failure, such as credit assignment, reward sparsity, or optimization instability? Are there additional experimental results that support this interpretation?

Q2. How sensitive are the experimental results to hyperparameter choices and algorithm adaptations? In particular, have the authors considered stronger exploration-oriented baselines or additional tuning for the continuing setting?

Reviewer_PcRVOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

This paper proposes a continual RL setting where an agent earns rewards for improving beyond its past performance. In standard RL, an agent can repeatedly earn rewards using a skill it has already mastered. This paper moves one step further: the world remembers the agent’s achievements and gradually stops rewarding repetition.

The proposed ArcadeWorld combines multiple environments into a single continuing lifetime. The agent chooses what actions to take within the current environment and whether to keep playing, restart, or switch environments. Each environment maintains a leaderboard of the agent’s prior scores. Rewards depend on performance relative to that history, encouraging the agent to improve its skills or move somewhere else where progress is feasible. This paper introduces two environments - BitWorld and AtariWorld. For BitWorld, after one million steps, standard deep RL methods—including DQN, PPO, Rainbow, and recurrent PPO underperform a simple rule-based agent that remembers 256 bits. For AtariWorld, random network distillation, trained using only intrinsic novelty rewards, performs better on several measures than agents trained on leaderboard rewards.

Critical strengths
  1. This paper addresses a useful research question is how an agent should allocate experience when familiar successes stop paying.
Critical weaknesses

This paper misses citations and comparisons to some very related work:

  1. Colas, Cédric, et al. "Curious: intrinsically motivated modular multi-goal reinforcement learning." International conference on machine learning. PMLR, 2019. https://proceedings.mlr.press/v97/colas19a/colas19a.pdf
  2. Graves, Alex, et al. "Automated curriculum learning for neural networks." international conference on machine learning. Pmlr, 2017. https://proceedings.mlr.press/v70/graves17a/graves17a.pdf
Questions for authors

Could you compare with the baselines below?

  1. Colas, Cédric, et al. "Curious: intrinsically motivated modular multi-goal reinforcement learning." International conference on machine learning. PMLR, 2019. https://proceedings.mlr.press/v97/colas19a/colas19a.pdf
  2. Graves, Alex, et al. "Automated curriculum learning for neural networks." international conference on machine learning. Pmlr, 2017. https://proceedings.mlr.press/v70/graves17a/graves17a.pdf
Reviewer_iawQOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper introduces ArcadeWorld, a framework that combines episodic environments into a single continuing interaction. The agent chooses both actions within an environment and when to reset or switch environments. Rewards depend on leaderboards of the agent’s previous scores, making past performance part of the underlying environment state. The contributions include a formal definition of leaderboard rewards, a characterization of rewards whose cumulative return preserves the ordering of final high scores, and a connection between reciprocal-placement rewards and count-based exploration. The paper also introduces BitWorld, a simple setting with interpretable memory-based baselines, and AtariWorld, which combines 57 Atari games. Experiments evaluate DQN, PPO, Rainbow, recurrent PPO, and exploration methods using count-based rewards or random network distillation. Standard deep RL methods fall well short of simple memory-based agents in BitWorld and fail to develop strong competence in AtariWorld.

Critical strengths
  1. ArcadeWorld makes past performance part of the environment state and uses it to determine future rewards. This provides a concrete framework for studying exploration, memory, and autonomous allocation of learning effort.

  2. The characterization of score-ordering rewards and the connection between reciprocal placement and visitation counts clarify what leaderboard rewards encourage. These results give the benchmark a principled foundation beyond an empirical task collection.

  3. BitWorld includes interpretable memory-based baselines and an oracle-observation variant, while experiments report uncertainty across multiple seeds. The authors also acknowledge that RND’s stronger record-setting performance does not translate into a substantially better final policy.

Critical weaknesses
  1. Historical maximum scores can improve through repeated attempts or occasional success without preserving learned skills. The limited final-policy evaluation does not establish broad competence or resistance to forgetting.
  2. AtariWorld sums improvements measured on different games’ native score scales. This may reward specialization in high-scoring games rather than the broad learning objective motivating the benchmark.
  3. The causes of baseline failure remain unclear. History-dependent rewards, shared-network interference, autonomous switching, and limited per-game experience are introduced together. Matched-budget controls with fixed scheduling and original rewards are needed to distinguish their effects.
  4. The policy-level implications of Proposition 3.2 are limited. The proof constructs an improved score sequence without establishing that a policy can realize it under the environment dynamics. The claim about incentives to pursue higher records therefore requires a clearer qualification
Questions for authors
  1. Could you report independent policy evaluations across all 57 games for all methods, using comparable cross-game metrics? Evaluations at multiple checkpoints would help establish whether agents acquire and retain skills rather than merely accumulate occasional record scores.

  2. Under the same total interaction budget, how do results change with fixed game scheduling or the original game rewards? Have you tested explicitly providing game identity, current score, and historical high score? These controls would help separate reward-history effects from task allocation, partial observability, and multi-task interference.

  3. The proof constructs a better score sequence by inserting a higher score at a future time, without establishing that this modification is realizable by an agent. Is the result intended only as a statement about abstract score sequences? What additional assumptions would support a corresponding claim about feasible policies?

  4. When the correct action is included in the observation, have you checked whether the same network can learn the mapping through a simple supervised control? This would help distinguish difficulties arising from RL credit assignment from basic representation or optimization issues

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.