- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
- PlayTrain is a toolchain that connects “LLMs writing games” with “RL training,” aiming to reduce the cost of developing of RL envs.
- The authors mainly use the existing Gemini 3.1 Pro, providing the game name, gameplay description, optional reference materials or source code, together with a fixed template, to generate JavaScript games.
- The generated games provide images, receive actions, and return rewards through a unified interface, serving as training environments where another RL agent learns through repeated play.
- The RL algorithms used are PPO and IMPALA, and the main technical contributions are the environment interfaces and a backend that bypasses the browser and uses native rendering and shared memory to improve execution efficiency.
- Across 24 cloned games, the system achieves a geometric mean of approximately 1.07 million agent decisions per second on a node equipped with four H100 GPUs, but complex tasks and generalization across game mechanics still lack sufficient validation.
Critical strengths
Check above summary.
Critical weaknesses
- Limited algorithmic innovation: Existing LLMs are used to generate games, and existing RL algorithms are used for training, with the contributions mainly concentrated in tools and systems engineering.
- Confounding factors in speed comparisons: The cloned games are not exactly the same as the originals, and some experiments also use different resolutions, so not all of the speedup can be attributed to the backend.
- Limited environment quality and scope: The experiments mainly validate simplified 2D games, and automated checks cannot guarantee correct rules, solvable levels, or rewards without loopholes, so human playtesting is still needed.
Questions for authors
None.
We thank the reviewers for their positive comments and helpful suggestions.
We thank the reviewer for this suggestion. To address these concerns, we validated complex tasks and generalization across game mechanics from a few standard benchmarks and made sure each are checked against the original:
We ported DMLab-30, 23 levels (3D navigation, memory and language) to PlayTrain. The ported levels match DMLab's layouts, movement and rewards. They run 1.3-29× faster than DMLab running natively on the same machine (0.8k–11k vs. 77–2,088 steps/s with one environment; new Table 22, Appendix G).
PlayTrain is also flexible to different mechanics, as it runs 26 VGDL games and 17 PuzzleScript puzzles, demonstrating a generalization across the original catalog of games we present in the paper (new Figure 20, new Table 22, Appendix G). All of these ports and games are available in the supplement to play and try-out.
We agree that PlayTrain is primarily a systems and tool contribution, not a new RL algorithm. However, by making novel environments inexpensive to build, PlayTrain will support future algorithmic work that is currently limited by the number and diversity of environments, such as meta-RL (as discussed in Section 4).
We thank the reviewer for this suggestion. To address the confounding speed concerns, we newly built JavaScript ports of five Atari games that exactly reproduce ALE’s game state to address the same-game backend measurements. All three engines produce the same 64×64 observations, which also removes the resolution difference the reviewer notes. When running 64 environments concurrently, our games, on average run 2.3x faster than Envpool (in their best tuned configuration). See the table below for a more detailed speed comparison. We add this comparison as new Table 16(a) (Appendix E.1).
Same-game speed:
All of the 5 Atari JS ports are playable and provided in the updated supplement.
We appreciate the reviewer for raising these critical points. We addressed each of these points as follows.
(1) To go beyond 2D games, we ported 23 3D DMLab levels to PlayTrain, which match the original and run 1.3–29× faster (new Table 22, new Appendix G).
(2) To catch some incorrect rules and rewards automatically, we added new automatic validator checks that play each game with random actions and test that controls work, rewards and episode ends are reachable, and resets are reproducible, and we tested them on injected and real bugs (new Tables 19–20, Appendix F). Lastly, to move beyond simplified 2D games.
(3) We regenerated all 24 games with Gemini 4 Argon with more detailed graphics (new Figure 19, Appendix F) and trained PPO and IMPALA on them (new Table 23, Figure 21, Appendix H) as additional experiments. All of these details are updated in the revised manuscript, and all newly presented games/ports are playable and trainable in the supplement.
We hope our responses clarify all the main concerns. Thank you very much for all the comments that help us improve our manuscript.