acceptodds
Under review as a conference paper at ICLR 2027

GameBoyWorlds: A Testbed for Self-Improvement in Embodied Video Games

Abstract

Powered by expert guidance, embodied agents can operate in interactive environments; however, it is unclear whether they can learn autonomously and continually from their own experience. To evaluate such self-improvement methods, we introduce a two-legged benchmark: , the first testbed for agentic self-improvement in complex video games. evaluates task execution on a collection of 5 distinct game series. Agents are allowed access to dedicated training games but are provided no demonstrations, documentation, or rewards. Agents must ground themselves in the environment through self-directed exploration and by inferring actionable knowledge from their own experience. At test time, agents must complete short-horizon execution tasks that evaluate their ability to navigate, interact, and engage with game-specific mechanics (e.g., combat) in potentially unseen games. Out-of-the-box frontier models complete fewer than 50% of the 500 tasks due to failures in multimodal grounding, establishing that self-improvement methods have considerable room to push performance. We demonstrate that contemporary approaches to self-improvement are lacking, with standard implementations of world modelling and autonomous skill discovery failing, and a novel strategy that uses curiosity-based exploration to write guides achieving only partial success. tests end-to-end game completion in two fan-made Pokémon games. We show that while frontier models have been pre-exposed to official releases such as Pokémon Red, they lack essential information on the games in our testbed. Instead of relying on their parametric knowledge to succeed, agents must learn from their own experience and autonomously improve over the course of the playthrough. We show that a sophisticated agentic pipeline with multimodal memory and hierarchical subgoals fails to reach even the first major milestone in both games, establishing GameBoyWorlds as an ambitious target for self-improving agents.

Primary area:
datasets and benchmarks
Keywords:
Self Improvement, Embodied Learning, Game Playing Agents, Continual Learning, Benchmark, Vision Language Model, Vision Language Action Models, VLA

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_Wm3cOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

Summary: This paper introduces GameBoy Worlds, the first benchmark platform designed to evaluate and foster autonomous self-improvement in embodied agents within complex video games. Traditional embodied AI heavily relies on human supervision, expert demonstrations, or hand-crafted reward signals. To bypass this bottleneck, GameBoy Worlds tests whether an agent can start with zero game-specific knowledge, demonstrations, or external rewards, and progressively acquire actionable skills and long-horizon capabilities purely through self-directed interaction and experience.

Contributions:

  • GameBoy Worlds-Execution (Short-Horizon Task Execution): A benchmark comprising 10 games across 5 distinct series and 500 hand-crafted tasks. Agents are given access to a training game per series without any explicit task examples or guidance, and must subsequently execute short-horizon tasks (navigation, combat, interaction) in test environments.
  • GameBoy Worlds-Playthrough (End-to-End Gameplay): A long-horizon challenge evaluating start-to-finish game completion across two fan-made Pokémon ROM hacks (Pokémon Brown and Pokémon Prism) designed to minimize pretraining data contamination and test true deployment-time learning.
  • Evaluation of Existing Paradigms: Systematic implementation and benchmarking of current self-improvement methods—including world modeling, autonomous skill discovery (PAE), and a novel curiosity-driven exploration pipeline that distills game insights into guidance documents.

Validation:

  • Execution Results: Out-of-the-box frontier models (such as Gemini-3.6-Flash, GPT-5-Mini, and Claude-4.5-Haiku) struggle significantly, achieving less than 50% average task completion due to multimodal grounding and perception failures. Standard self-improvement methods either fail or cause catastrophic performance degradation (e.g., skill discovery dropping performance by over 40%), while the proposed curiosity-to-documentation pipeline yields only modest and inconsistent gains (+2.0% average), proving the execution leg remains largely unsolved.
  • Playthrough Results: Through rigorous contamination checks using 400 questions, the authors confirm that while models possess high parametric knowledge of official games like Pokémon Red, they lack procedural knowledge for the fan-made hacks (<30% accuracy). When deployed on the playthrough challenge, an advanced agentic pipeline with hierarchical subgoals and memory successfully navigated early milestones in contaminated official games but completely failed to clear even the first major milestone (obtaining the starter Pokémon) in the unfamiliar fan-made games.
Critical strengths
  • Novel Benchmark Design for Self-Improvement: The introduction of GameBoy Worlds provides a rigorous and much-needed testbed to evaluate autonomous, zero-shot self-improvement in embodied agents without relying on human demonstrations, hand-crafted external rewards, or expert priors. This addresses a critical bottleneck in scaling embodied AI.

  • Rigorous Handling of Data Contamination: The inclusion of the Playthrough leg utilizing fan-made ROM hacks (Pokémon Brown and Pokémon Prism) alongside systematic contamination checks effectively isolates true deployment-time learning and active knowledge acquisition from mere memorization of pretraining corpora.

Critical weaknesses
  • Limited Novelty in Methodological Outcomes: Despite introducing a comprehensive benchmark, the evaluated self-improvement methods (world modeling, skill discovery, and curiosity-guided documentation) either fail catastrophically or yield only marginal, inconsistent gains, leaving the benchmark largely unsolved without providing a robust, generalizable algorithmic solution to overcome these failure modes.

  • Evaluation Constraints Driven by Emulator Parsing: Relying on screen-region captures and visual state comparisons restricts task verification to events with immediate visual feedback, potentially filtering out complex, long-horizon internal state transitions that lack distinct graphical representations.

Questions for authors
  • Impact of Verification Bottlenecks on Task Distribution: In Appendix A.5, the authors note that task verification relies on immediate visual feedback (e.g., distinct screen changes or dialogue boxes), which inherently excludes long-horizon internal state transitions. Could the authors clarify how severely this constraint skews the execution benchmark away from complex, non-visual strategic progress (such as inventory management or hidden status modifications)? Furthermore, what fraction of potential tasks had to be discarded due to this limitation?

  • Generalization vs. Overfitting in Curiosity-Based Exploration: While the curiosity-to-documentation pipeline achieved modest gains on certain games, it resulted in severe performance degradation on others (e.g., -29% on Deja Vu 1). Can the authors clarify whether the negative transfer is primarily caused by noisy insight extraction from the VLM or by the inherent brittleness of PPO-discovered trajectories when applied across different task distributions?

Reviewer_us1EOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes the GameBoyWorlds benchmark to evaluate the existing self-improvement methods of AI agents in interactive environments. The benchmark consists of a short-horizon leg GameBoyWorlds-Execution and a long-horizon leg GameBoyWorlds-Playthrough. The short-horizon leg tests agents' task execution ability after self-driven exploration. The authors evaluate two existing self-improvement methods (world modeling and autonomous skill discovery) along with their own curiosity-based method on it, and find that none consistently improves performance. The long-horizon leg tests models on two fan-made Pokémon variants, where the authors' agent fails to obtain even the first Pokémon.

Critical strengths
  • The paper is clear about the benchmark motivation, construction, and formulation. Its definition of self-improvement is rigorous.
  • The GameBoyWorlds-Execution benchmark tests generalization at two levels: agents apply what they learn through self-driven exploration to unseen tasks in the same game as well as an unseen game in the same familiy.
  • The paper uses two fan-made Pokémon variants as the Playthrough benchmark to mitigate benchmark contamination. It supports this choice with an agent comparison against the original Pokémon Red and Crystal.
Critical weaknesses
  • Models are inconsistent across experiments. The Execution baseline (Figure 2) uses five models, but self-improvement methods in Section 4 are only tested on Gemma-4-31B. The contamination check in Figure 6 uses GPT-5.6-sol, Claude Opus 5 and Gemini 3.5 Flash-Lite, but the reported playthrough (in Figure 7) uses Gemini-3.6-Flash.
  • The Playthrough benchmark evaluation is limited. The paper only documents a supervisor-strategist agent with one model without any self-improvement methods evaluated. Results are only a qualitative description within the first 100 agent loops.
  • Contamination is not addressed in the Execution benchmark. Pokémon Red and Crytal are in that benchmark but the paper does not discuss how prior knowledge affects the measured self-improvement.
  • Related work is not fully discussed. Section 6 covers only frontier-model Pokémon demos and ARC-AGI-3. Benchmarks cited in the introduction (such as BALROG, VideoGameBench, and NetHack) are not compared with. The 99.9% score on ARC-AGI-3 is discussed but has no citation.
Questions for authors
  • Could the self-improvement methods be evaluated on additional base models? Now it is unclear whether the existing self-improvement methods fail across models, and how model ability affects the gains from self-improvement.
  • How does contamination affect the Execution benchmark?
  • Could the authors define an evaluation protocol for the Playthrough benchmark (interaction budget, metric, and allowed scaffolding) and report quantitative results for more agents and models? Currently the Playthrough results are a qualitative description of a single agent with one model.

Overall, I find the problem setting important and the benchmark design valuable, but the Playthrough conclusions rest on a single model, a single agent, and roughly the first 100 agent loops. I would be willing to raise my score if the authors evaluate the Playthrough benchmark with a longer interaction budget and at least one additional model and agent.

Reviewer_CLYDOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

GameBoyWorlds benchmarks includes 500 short tasks across ten games and a playthrough test on Pokemon Brown and Prism. Agents explore training games without demonstrations, guides, or task-specific rewards, then face tasks in both familiar and unseen games. The authors test world modeling, autonomous skill discovery, and guides generated from curiosity-driven exploration. None improves performance consistently across games. The guide method raises average success from 40.7% to 41.5%. Knowledge checks show that models know much less about the fan-made games than the official versions. The tested playthrough agent does not obtain its first Pokemon in either fan-made game.

Critical strengths
  1. The benchmark separates short-task execution from learning during a full playthrough and tests transfer to unseen games in the same series. Using less documented fan-made games also reduces the influence of memorized walkthroughs.

  2. The tasks cover navigation, menus, interaction, and combat across five game series. Deterministic resets and frame-based success checks make evaluation reproducible, while human validation confirms that the tasks are achievable. Success is checked by the environment rather than the agent itself.

Critical weaknesses
  1. The experiments show that the tested methods struggle, but do not clearly explain why. The world model lacks quantitative prediction checks, and the proposed explanation for skill-discovery failures is not verified through an audit of self-evaluation errors. The guide method improves success from 40.7% to 41.5%, without uncertainty estimates or a matched-budget random-exploration control. It is therefore unclear whether the gain is reliable or comes from curiosity-driven exploration.

  2. The playthrough results depend on one model and one agent design, with run counts, budgets, and stopping criteria insufficiently reported. Lower knowledge-check scores on Brown and Prism do not establish why gameplay fails; exploration and control may also contribute. Several prompts in Appendix E.7 explicitly name Pokemon Red, and the authors should clarify whether these were used unchanged in the fan-made games. The results support a narrower conclusion about the tested system rather than a general limitation of deployment-time learning.

Questions for authors
  1. How accurate are the world model's action predictions? Have you manually checked whether trajectories labeled successful during skill discovery actually complete the proposed tasks?

  2. Is the guide method's 0.8-percentage-point gain consistent across repeated runs? How does it compare with random exploration under the same budget, and does it help both training games and unseen sequels?

  3. How many playthrough runs were conducted, with what action and model-call budgets and stopping criteria? Does a correct exploration hint help the agent leave the starting town?

  4. Several prompts in Appendix E.7 name Pokemon Red. Were these used unchanged in Brown and Prism? If so, do the failures persist after correcting the game name?

Comments on the paper

1 comment
AuthorsOpenReview ↗

Summary of revisions, and request for engagement

We thank all the reviewers for their comments. As highlighted in the individual responses below, here are the full summaries of how we have revised the paper and added concrete experiments to address the reviewers' concerns:

Reviewer CLYD

Weakness 1: The experiments show that the tested methods struggle, but do not clearly explain why.

Response: We respond with concrete quantitative comparisons for each method.

We show:

World Models:

  • Pixel-space loss is likely insufficient.
  • World models struggle under distribution shift.
  • We isolate the gains from world modelling vs planning and show that both must be improved.

Autonomous Skill Acquisition:

  • We show with human annotation that the pipeline fails at self-evaluation
  • We show that the same pipeline does not fail in Web domains, establishing the unique difficulty of self-evaluation in GameBoyWorlds
  • We conduct an analysis of the proposed tasks and establish that they cover the test tasks, showing that task proposal is not the core issue.

Curiosity-based guidance documents:

  • We show with budget-matched controls that the curiosity-driven exploration process adds genuine value, and that it is a viable algorithm for future study, joining the likes of world modelling and autonomous skill acquisition.

This also answers (Question 1 and 2)

Weakness 2: The playthrough results depend on one model and one agent design, with run counts, budgets, and stopping criteria insufficiently reported.

Response:

  • We run on additional models, and show results for additional agent designs
  • We provide run counts, budgets, stopping criteria and more descriptive progress tracking information
  • We show with a controlled information injection experiment that control is not the bottleneck, but exploration is.

Reviewer us1E

Weakness 1: Models are inconsistent across experiments

Response:

  • We have patched this inconsistency and show results for consistent models when possible (we cannot fine-tune closed-source models, hence those are out-of-scope for approaches such as PAE)

Weakness 2: The Playthrough benchmark evaluation is limited

Response:

  • Added results for more models and agent paradigms.
  • Reported more evaluation metrics, e.g. number of unique tiles discovered.
  • Directly addressed stopping criteria and budget, etc.

Weakness 3: No discussion on contamination in execution benchmark

Response: Showed that contamination is not the primary determinant of performance in the execution benchmark (unlike with the Playthrough benchmark) with multiple experiments.

Weakness 4: Related work lacks discussion

Response: Discussed related work in more detail and added citations

Question 1: Can Self-Improvement be run on other models

Response: Done, results are consistent across 3 models.

Question 2: How does contamination affect the Execution benchmark?

Response: Answered with Weakness 3. Contamination is not the primary determinant of performance in GameBoyWorlds-Execution

Question 3: Could the authors define an evaluation protocol for the Playthrough benchmark

Response: Yes. Done above in response to weakness 2.

Reviewer Wm3c:

Limited Informativeness:

Response: We have provided a list of insights we think one may draw from our work

Weakness 1: All self-improvement methods fail. Limited methodological novelty.

Response: Stressed that the failure of standard approaches is a meaningful finding, and we shed light on which parts of each pipeline goes wrong.

Weakness 2: Limited task distribution

Response: Clarified the limitation regarding task distribution. We can and do support long-horizon tasks.

Ethics Review:

Response: Ethics Statement added to the paper. The statement explicitly discusses the risks of copyright and provides information to help researchers engage in more responsible research practices.

We have made an effort to respond to all of the reviewers comprehensively, with additional experiments when possible. Seeing how the discussion period is nearly over, we ask the reviewers to let us know whether we have answered their questions and concerned, and if not, give us a chance to follow up.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.