acceptodds
Under review as a conference paper at ICLR 2027

PlayTrain: An Efficient Reinforcement Learning Framework for LLM-Generated Adaptable JavaScript Games

Abstract

While many video-game environments (VGEs) have played crucial roles in advancing reinforcement learning (RL), developing novel VGEs or modifying existing ones to support new features, has been a laborious process requiring extensive hand-coding. Here we present PlayTrain, an RL framework that combines the abilities of large language models (LLMs) to robustly generate JavaScript (JS) games from a minimal human prompt, and an efficient pipeline that can run any JS game in a standard 'gym' environment. Not only are recent LLMs particularly good at writing JS code, but the JS format also allows users to easily play generated VGEs, while PlayTrain enables us to train RL agents on the exact same games. We demonstrate multiple use cases of PlayTrain, including cloning well-known Atari and ProcGen games in simple JS, where PlayTrain trains pixel-based agents end-to-end at over 1M agent-decisions per second on a single GPU node; and creating modified versions thereof (e.g., that support novel test sets, procedural generation logics, or game dynamics). Through PlayTrain, we reimagine RL VGE development: all we need is a single JS file, generated and modified through an LLM. We discuss promising future RL research directions that PlayTrain unlocks.

Primary area:
reinforcement learning
Keywords:
reinforcement learning, environments, LLM code generation

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_KzUKOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary
  1. PlayTrain is a toolchain that connects “LLMs writing games” with “RL training,” aiming to reduce the cost of developing of RL envs.
  2. The authors mainly use the existing Gemini 3.1 Pro, providing the game name, gameplay description, optional reference materials or source code, together with a fixed template, to generate JavaScript games.
  3. The generated games provide images, receive actions, and return rewards through a unified interface, serving as training environments where another RL agent learns through repeated play.
  4. The RL algorithms used are PPO and IMPALA, and the main technical contributions are the environment interfaces and a backend that bypasses the browser and uses native rendering and shared memory to improve execution efficiency.
  5. Across 24 cloned games, the system achieves a geometric mean of approximately 1.07 million agent decisions per second on a node equipped with four H100 GPUs, but complex tasks and generalization across game mechanics still lack sufficient validation.
Critical strengths

Check above summary.

Critical weaknesses
  1. Limited algorithmic innovation: Existing LLMs are used to generate games, and existing RL algorithms are used for training, with the contributions mainly concentrated in tools and systems engineering.
  2. Confounding factors in speed comparisons: The cloned games are not exactly the same as the originals, and some experiments also use different resolutions, so not all of the speedup can be attributed to the backend.
  3. Limited environment quality and scope: The experiments mainly validate simplified 2D games, and automated checks cannot guarantee correct rules, solvable levels, or rewards without loopholes, so human playtesting is still needed.
Questions for authors

None.

1 comment

We thank the reviewers for their positive comments and helpful suggestions.

Across 24 cloned games, the system achieves a geometric mean of approximately 1.07 million agent decisions per second on a node equipped with four H100 GPUs, but complex tasks and generalization across game mechanics still lack sufficient validation.

We thank the reviewer for this suggestion. To address these concerns, we validated complex tasks and generalization across game mechanics from a few standard benchmarks and made sure each are checked against the original:

  • We ported DMLab-30, 23 levels (3D navigation, memory and language) to PlayTrain. The ported levels match DMLab's layouts, movement and rewards. They run 1.3-29× faster than DMLab running natively on the same machine (0.8k–11k vs. 77–2,088 steps/s with one environment; new Table 22, Appendix G).

    • Importantly, the original DMLab environments are currently very difficult to run on a modern Linux system (our builds on Ubuntu 22.04 repeatedly failed) as they are pinned to deprecated libraries. We see this as a highlight for PlayTrain’s simplicity.
  • PlayTrain is also flexible to different mechanics, as it runs 26 VGDL games and 17 PuzzleScript puzzles, demonstrating a generalization across the original catalog of games we present in the paper (new Figure 20, new Table 22, Appendix G). All of these ports and games are available in the supplement to play and try-out.

(W1) Limited algorithmic innovation: Existing LLMs are used to generate games, and existing RL algorithms are used for training, with the contributions mainly concentrated in tools and systems engineering.

We agree that PlayTrain is primarily a systems and tool contribution, not a new RL algorithm. However, by making novel environments inexpensive to build, PlayTrain will support future algorithmic work that is currently limited by the number and diversity of environments, such as meta-RL (as discussed in Section 4).

(W2) Confounding factors in speed comparisons: The cloned games are not exactly the same as the originals, and some experiments also use different resolutions, so not all of the speedup can be attributed to the backend.

We thank the reviewer for this suggestion. To address the confounding speed concerns, we newly built JavaScript ports of five Atari games that exactly reproduce ALE’s game state to address the same-game backend measurements. All three engines produce the same 64×64 observations, which also removes the resolution difference the reviewer notes. When running 64 environments concurrently, our games, on average run 2.3x faster than Envpool (in their best tuned configuration). See the table below for a more detailed speed comparison. We add this comparison as new Table 16(a) (Appendix E.1).

Same-game speed:

GameALEEnvPoolPlayTrainvs. EnvPool
breakout9.8k13.0k30.3k2.32×
freeway8.6k10.5k99.2k9.48×
frostbite10.1k12.2k16.1k1.31×
pong13.1k16.0k41.2k2.57×
space_invaders11.0k12.8k12.0k0.94×
Geometric mean10.4k12.8k29.9k2.34×

All of the 5 Atari JS ports are playable and provided in the updated supplement.

(W3) Limited environment quality and scope: The experiments mainly validate simplified 2D games, and automated checks cannot guarantee correct rules, solvable levels, or rewards without loopholes, so human playtesting is still needed.

We appreciate the reviewer for raising these critical points. We addressed each of these points as follows.

(1) To go beyond 2D games, we ported 23 3D DMLab levels to PlayTrain, which match the original and run 1.3–29× faster (new Table 22, new Appendix G).

(2) To catch some incorrect rules and rewards automatically, we added new automatic validator checks that play each game with random actions and test that controls work, rewards and episode ends are reachable, and resets are reproducible, and we tested them on injected and real bugs (new Tables 19–20, Appendix F). Lastly, to move beyond simplified 2D games.

(3) We regenerated all 24 games with Gemini 4 Argon with more detailed graphics (new Figure 19, Appendix F) and trained PPO and IMPALA on them (new Table 23, Figure 21, Appendix H) as additional experiments. All of these details are updated in the revised manuscript, and all newly presented games/ports are playable and trainable in the supplement.

We hope our responses clarify all the main concerns. Thank you very much for all the comments that help us improve our manuscript.

Reviewer_veJBOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

The paper presents PlayTrain, a framework for creating RL enviroments based on Javascript generated using a LLM. The LLM writes a single JS file based from a user prompt specifying the game description, followed by verification and game refinement. The authors make 24 clones of Atari and ProcGen games, plus a few variants and two new games. They train PPO and IMPALA agents on these games and compare the agents with human players.

The paper makes the following contributions: - A pipeline where an LLM generates RL games from a short text description. - A browserless backend that runs the games inside the training process. It reports over 1M agent steps per second - Learning curves for PPO and IMPALA on all 24 cloned games, and a small human study comparing human and agent scores on 8 games.

Critical strengths
  • The idea is useful and easy to understand. RL environments can be described in plain text, generated by an LLM.

    • Play testing the game is easier, since the games are JS based and can be run on a browser.

    • The authoirs are honest about the costs and limitations of there method.

Critical weaknesses
  • Generation reliability is not evaluated. The automatic validation checks only format and determinism (Table 16), not whether the game logic is are correct. The paper does not report how much refinement the 24 clones needed.

    • Several agents score close to or below a random policy, and the paper does not discuss this. Examples from Table 10: IMPALA on jumper (0.3 vs 0.8 random), maze (1.7 vs 3.8) and leaper (12.0 vs 12.5).

    • The text states Table 10 shows 95% confidence intervals, but the table contains only means.

    • The reported throughput differes between Table 1 and Table 9, even though it appears to be same configuration (IMPALA, Nature-CNN, all 24 games).

    • Writing could be better. The text contains many typos ("vecotrized", "divded", "intead", "meausre", "the the"). Figure and Table references should be checked (Refereced Figure 8B should be 8C)

Questions for authors
2 comments
AuthorsOpenReview ↗

Response to Reviewer veJB (1/2)

We thank the reviewers for their overall positive comments and constructive/actionable questions.

Generation reliability is not evaluated. The automatic validation checks only format and determinism (Table 16), not whether the game logic is are correct.

We thank the reviewers for this point. In response, we added extra checks to our automatic validation checker:

  • Behavioral checks. We roll out each game under a random policy and check that observations are different, resets with the same seed are reproducible, actions affect the game state, and rewards and episode ends are reachable.
  • Cross-engine check. We make sure each game reproduces the same observations across the QuickJS engine and the standard V8 JS engine.
  • Automated repair. Given these new validation checks, we also allow for three “fix” attempts based only on the validator’s output.
  • VLM evaluation (tested separately). For a more robust addition, we also tested a vision-language model to assess key events within the game (e.g. resets, rewards, life losses). The VLM simply sees these key-frames and captions issues it finds.

We also conducted specific experiments to test whether these extra checks are effective by letting it (1) detect artificially injected bugs into our original 24 games and (2) finding natural bugs we encountered during the original generation.

Validator evaluation:

Bug setBugsOriginal validatorNew validator
Artificial injected bugs (e.g. swapped controls, a reward that never fires)23018%82%
Natural bugs from our refinement logs142 (14%)8 (57%)

Most missed bugs are semantic such as wrong reward values or hitboxes. We report these results in new Table 20 (Appendix F), and the new checks are also listed in new Table 19.

All code and logs for the newly updated automatic validator and experiments are provided in the updated supplement.

The paper does not report how much refinement the 24 clones needed.

Thank you for this suggestion. Among the initial 24 games, 20 generated games required no refinement. The remaining four games (jumper, qbert, climber, and freeway) needed 3.5 refinement API calls on average. We did not record token counts for these games.

To address the lack of initial logs, we regenerated the 24 games with Gemini 4 Argon (with more complex visuals) and logged every model call, token and human edit (table below). You can find this in the updated manuscript (new Table 21, Appendix F). All regenerated games can be played within the supplement. (For the sake of completeness, we also repeated all the training experiments on these new games and reported them as additional results in Appendix H.)

Regeneration effort:

StageModel callsGamesCostHuman time
Generation2424$2.66–
Automated fixes118$1.41–
Human refinements2611$2.252.4 min each
Total6124$6.3276 min total
AuthorsOpenReview ↗

Response to Reviewer veJB (2/2)

Several agents score close to or below a random policy, and the paper does not discuss this. Examples from Table 10: IMPALA on jumper (0.3 vs 0.8 random), maze (1.7 vs 3.8) and leaper (12.0 vs 12.5).

Thank you for pointing this out. In order to address this, (1) we adopted epsilon-greedy instead of greedy evaluation, and (2) increased the number of evaluation environments (from 8 to 32). Both helped as shown on the table below (new Table 11, Appendix D). Agents at or below random:

AgentRandom (8 / 32 levels)Greedy, 8 levelsGreedy, 32 levelsε-greedy, 32 levelsWhat greedy does
IMPALA maze3.8 / 5.31.71.1 ± 2.76.1 ± 2.0Runs into a wall (89%)
PPO maze3.8 / 5.32.92.7 ± 0.47.5 ± 2.1Runs into a wall (73%)
IMPALA jumper0.8 / 0.70.30.5 ± 0.46.5 ± 0.6Repeats a jump loop
PPO caveflyer1.6 / 2.01.11.3 ± 1.71.8 ± 1.0Crashes early
PPO asteroids513.8 / 574.7479.2506.5 ± 502.0615.1 ± 460.5One seed only fires
IMPALA leaper12.5 / 11.512.013.3 ± 15.214.1 ± 11.0Above random
PPO climber0.2 / −0.5−0.1−0.1 ± 0.4−0.1 ± 0.4Above random

The only exception is PPO on caveflyer, at 1.8 against random's 2.0, within the confidence interval. PPO never sharpened its policy on this game, so under arg-max it holds a single action and crashes into the cave walls.

The text states Table 10 shows 95% confidence intervals, but the table contains only means.

The reported throughput differes between Table 1 and Table 9, even though it appears to be same configuration (IMPALA, Nature-CNN, all 24 games).

Writing could be better. The text contains many typos ("vecotrized", "divded", "intead", "meausre", "the the"). Figure and Table references should be checked (Refereced Figure 8B should be 8C)

Thank you for the careful reading.

  • Table 10. We now report 95% CIs over the three training seeds, computed from the existing per-seed evaluations.
  • Table 1 vs Table 9. These were different setups: 12 workers in Table 1, and 15 workers on a different node in Table 9. We now state the setup in both captions.
  • Typos and references. All fixed, including Figure 8B → 8C, and we checked every reference.

We hope our responses clarify all the main concerns.

Thank you very much for all the comments that help us improve our manuscript.

Reviewer_QufnOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

PlayTrain couples LLM-generated single-file JavaScript games with an efficient RL backend. QuickJS, a C++ subset of p5 drawing functions, and a Rust rasterizer generate observations directly into shared buffers. The same game source can be played in a browser and used for RL. Experiments cover Atari/ProcGen-inspired replicas, game variants, learning curves, throughput comparisons, and an illustrative human study.

Critical strengths

A practical combination of editable games, fast pixel observations, and immediate human playability. Efficient engineering integrates execution, drawing, rasterization, and shared-memory collection without browser overhead. Throughput is decomposed across backends and training architectures, and slowdown cases are explicitly discussed. Runnable variations and actual learning curves demonstrate that the contribution goes beyond rendering toy scenes.

Critical weaknesses

The paper acknowledges that clones differ in visuals/details. Faster execution than original C++ games therefore cannot establish a pure backend advantage. Keep the same-game backend measurements central and quantify mechanics/state complexity for replica comparisons. One-shot API/determinism success is distinct from faithful game generation; the paper allows human play-testing and refinement. Report all initial attempts, correction rounds, human time, API cost, and semantic failure categories. The custom p5 implementation supports a subset of drawing operations and a prescribed lifecycle. Running identical source files does not guarantee identical observations or dynamics across browser and custom execution. A supported-feature matrix and cross-backend action-trace parity tests would establish the boundary. The variants show feasibility, but a held-out mutation suite with success rates and a transfer/generalization experiment would better support the proposed research benefits. This need not require a new RL algorithm.

Questions for authors

Can the strategy generalized to any other type of game generation?

2 comments
AuthorsOpenReview ↗

Response to Reviewer Qufn (1/2)

We thank the reviewers for their valuable feedback and positive comments.

Faster execution than original C++ games therefore cannot establish a pure backend advantage. Keep the same-game backend measurements central and quantify mechanics/state complexity for replica comparisons.

We thank the reviewer for this suggestion. To address this, we built JavaScript ports of five Atari games that exactly reproduce ALE’s game state to address the same-game backend measurements. When running 64 environments concurrently, our games, on average run 2.3x faster than Envpool (in their best tuned configuration).

We also ran ProcGen's own C++ games through our harness, where it is slightly faster than EnvPool (1.03–1.25×; new Table 16(b), Appendix E.1). See the table below for a more detailed speed comparison. We add this table as new Table 16(a) (Appendix E.1).

Concerning mechanics/state complexity, Figure 15 and Table 17 (Appendix E.2) split each game’s per-step cost into drawing and game logic.

Same-game speed:

GameALEEnvPoolPlayTrainvs. EnvPool
breakout9.8k13.0k30.3k2.32×
freeway8.6k10.5k99.2k9.48×
frostbite10.1k12.2k16.1k1.31×
pong13.1k16.0k41.2k2.57×
space_invaders11.0k12.8k12.0k0.94×
Geometric mean10.4k12.8k29.9k2.34×

All of the 5 Atari JS ports are playable and provided in the updated supplement.

One-shot API/determinism success is distinct from faithful game generation; the paper allows human play-testing and refinement. Report all initial attempts, correction rounds, human time, API cost, and semantic failure categories.

Thank you for this suggestion. Among the initial 24 games, 20 generated games required no human refinement. The remaining four games (jumper, qbert, climber, and freeway) needed 3.5 human refinements on average (14 refinements in total). Unfortunately, we did not record token counts for these games.

To address the lack of initial logs, we regenerated the 24 games with Gemini 4 Argon (with more complex visuals) and logged every model call, token and human edit (table below). You can find this in the updated manuscript (new Table 21, Appendix F). All regenerated games can be played within the supplement. (For the sake of completeness, we also repeated all the training experiments on these new games and reported them as additional results in Appendix H.)

Regeneration effort:

StageModel callsGamesCostHuman time
Generation2424$2.66–
Automated fixes118$1.41–
Human refinements2611$2.252.4 min each
Total6124$6.3276 min total
AuthorsOpenReview ↗

Response to Reviewer Qufn (2/2)

The custom p5 implementation supports a subset of drawing operations and a prescribed lifecycle. Running identical source files does not guarantee identical observations or dynamics across browser and custom execution. A supported-feature matrix and cross-backend action-trace parity tests would establish the boundary.

Thank you for the suggestion. To address this, (1) we replayed the same action traces (38 games, 10 seeds × 3,000 random actions) in Chromium, WebKit and Firefox under the original p5.js. We found that rewards, episode terminations, scores and lives match PlayTrain in all 380 cases, while frames differ in about 1% of pixels, almost all on the edges of drawn shapes (new Table 5b, Figure 10, Appendix C.1). (2) The two engines we present in the paper (QuickJS and Node.js) give identical frames, and we updated our validator to check this for every newly created game. Regarding a supported-feature matrix, we newly added Table 5 (Appendix C.1) which now lists which p5.js commands PlayTrain supports and which it does not.

The variants show feasibility, but a held-out mutation suite with success rates and a transfer/generalization experiment would better support the proposed research benefits.

Thank you for pointing this out. We have ran transfer experiments on five generated variants of four games (Breakout, Qbert, Frostbite, Seaquest). Agents trained on the base game mostly drop to random-level performance on the variants zero-shot, even on a purely visual re-skin of Frostbite. However, fine-tuning from the base game beats training from scratch for IMPALA on every pair at 25M steps (e.g., 815 vs. 303 on a Qbert variant, 593 vs. 7 on a Seaquest variant). We added a new Table 12 (Appendix D) to report these results. We agree a held-out mutation suite with success rates would strengthen this, and we will add one in the final version.

Can the strategy generalized to any other type of game generation?

Yes, this strategy generalizes to other kinds of games written in JavaScript. As an example, we ported DeepMind Lab 3D game environments to PlayTrain, by simply using Claude Code with tests checking to match the DMLab’s game-state, as evidence for this strategy generalization, which is newly added in Appendix G. As LLM agents make web games easier to generate, our pipeline can use any JavaScript game library, and it already produces physics games (Matter.js, Appendix A).

We hope our responses clarify all the main concerns. Thank you very much for all the comments that help us improve our manuscript.

Comments on the paper

2 comments

We thank all the reviewers for valuable feedback. We have uploaded the revision in the PDF by incorporating all the new results introduced in the rebuttal to address reviewers’ comments. Changes are highlighted in blue. All of these changes are also reflected in the revised supplemental. The summary of new tables/figures is as follows:

  • Figure 6, More environments: We have added more environments to further demonstrate PlayTrain’s capabilities and highlight it in Figure 6 (DMLab-30, PuzzleScript, Craftax-Classic, and visually enhanced Atari/ProcGen clones), demonstrating flexibility in what PlayTrain generates and trains.

  • Table 16 (Appendix E.1), Speed confound experiment: We built JavaScript ports of five Atari games that reproduce ALE’s exact game state and compared it with ALE and EnvPool on the original ALE ROMs (Reviewer Qufn and KzUK).

  • Figure 20 & Table 22 (Appendix G), Additional ports of existing games: To demonstrate the generality of PlayTrain beyond simple 2D games, we’ve added 23 DMLab-30 levels, 17 PuzzleScript games and 26 VGDL games. Each of these ports, and the five Atari ports mentioned above, were built simply by using Claude Code with tests checking that their game state matched the originals. (Reviewer KzUK and Qufn)

  • Table 21 (Appendix F), Generation logging: We regenerated 24 games (16 ProcGen, 8 ALE) with more complex graphics using Gemini 4 Argon and reported every API call, token amount, cost, and time of playtesting. (Reviewer Qufn and veJB)

  • Table 23 & Figure 21 (Appendix H), Training on the regenerated games: We report results of IMPALA and PPO on all of the newly generated 24 games.

  • Tables 19–20 (Appendix F), Updates to our automatic validator: We added additional behavioural and cross-engine checks to our validator and more specific automated repair. This validator is tested on injected and real bugs (Reviewer veJB and KzUK)

  • Table 10–12 (Appendix D), Confidence intervals: We have added the 95% confidence intervals in the newly made Table 10 and a re-evaluation of the agents who scored below random. We also ran transfer generalization experiments on the generated variants. (Reviewer veJB)

  • Table 5 (Appendix C.1), Browser parity and the supported p5.js subset: We ran our games across three different browsers (Chromium, WebKit and Firefox) under the original p5.js and confirmed that rewards, episode ends, scores and lives match PlayTrain's in all 380 runs. We also list which p5.js commands PlayTrain supports. (Reviewer Qufn)

We hope these updates address the main concerns raised from the reviewers, and strengthen the overall demonstration of PlayTrain’s capabilities. Thank you very much again for helpful suggestions.

The newly made visually enhanced replicas (16 ProcGen environments, 8 ALE environments) still run, on average, faster than the original replicas despite their richer graphics. The table below presents their speeds: 1.17× the speed on a single CPU core, and the same training throughput.

Geometric mean over 24 gamesOriginal replicasVisually enhanced replicasEnhanced / original
Environment steps/s, single CPU core26.9k31.5k1.17×
Environment steps/s, 80 CPU threads2.81M3.02M1.08×
Training agent-steps/s, IMPALA296k309k1.04×
Training agent-steps/s, PPO13.9k13.9k1.00×

Both replica sets were measured on a single core of the same Intel Xeon Platinum 8480+ machine and on 80 threads of the same AMD EPYC 9654 machine, and trained with the same setup described in the paper.

We promise to add these numbers, with more detail, to the final version of the manuscript. The supplement is also updated with the raw measurements and the code that reproduces these numbers.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.