- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper presents RPG Dreamer, an agent-based framework for automatically generating long-horizon role-playing games (RPGs). The two main challenges in existing coding agents are the entanglement of game mechanics and content, and insufficient capabilities for effective game testing. To address these challenges, the paper introduces two components: the Agent-Oriented Game Engine (AOGE) and the Reviewer-centric Generation Loop (RGL). AOGE separates reusable game mechanics from game-specific content and provides a unified Agent Interface. RGL uses code, content, and gameplay reviewers to iteratively inspect and refine generated games. The authors also introduce RPG-Bench, comprising 150 game-generation tasks. The benchmark evaluates generated games through dynamic gameplay trajectories along four dimensions: requirement fidelity, content richness, visual and interaction quality, and playability. Experiments show that RPG Dreamer substantially outperforms existing coding agents and game-generation approaches across several metrics.
Critical strengths
- Current coding agents struggle with complex narratives, branching quests, and long-horizon state dependencies. This paper focuses on addressing this challenge by focusing on complete games that can be played over extended periods, rather than short demos.
- Dynamic evaluation based on actual gameplay trajectories can reveal inconsistencies in game logic that static evaluations may overlook.
Critical weaknesses
- Completion rate alone does not adequately support the motivation that game-generation quality deteriorates with complexity. In Section 2.2, the authors observe that route completion rates decrease substantially as game complexity increases from L1 to L5. However, completion rate is not a direct measure of generation quality. A breakdown of unsuccessful routes by failure type would provide stronger support for the stated motivation.
- The manually constructed game mechanics create a gap between the claimed task of complete game generation and what the framework actually generates. The paper emphasizes generating complete, complex, long-horizon games from natural-language requirements. However, Section 3 and Appendix F.1 indicate that the core mechanics in AOGE are manually implemented in advance and remain fixed during subsequent generation. The task therefore appears closer to automatic game-content generation on top of a pre-engineered framework than fully end-to-end game generation. Since game mechanics and their interactions are central to game design and playability, providing them in advance may substantially reduce task difficulty. This also raises concerns about the fairness of the baseline comparison where RPG Dreamer benefits from prebuilt, tested mechanics, whereas baseline coding agents must implement corresponding systems themselves. The authors should clarify the scope of generation supported by the framework and ideally compare methods under the same prebuilt-mechanics setting.
- The playability metric primarily measures functional correctness rather than the broader quality of gameplay. Playability (PL) receives a weight of 30% in the overall score. However, according to Section 4.2 and Appendix G.1, it is largely assessed through quest completion, game-state consistency, and basic functional operations. These measures capture functional correctness and progression reliability, but do not fully represent playability. This concern is particularly relevant given that the main game mechanics are manually predefined. The authors should more clearly delimit what PL measures and, if possible, provide complementary evidence on gameplay quality and real playtime.
Questions for authors
Q1. In the complexity-scaling experiment in Section 2.2, could the authors provide a breakdown of why routes were not completed? How many failures were caused by implementation bugs, as opposed to legitimate gameplay difficulty or insufficient guidance? Q2. Which mechanics in AOGE are manually implemented? If a user requests a new gameplay mechanic or a game system not supported by the current AOGE, can RPG Dreamer autonomously design and implement it? Q3. If all baselines were provided with the same prebuilt game mechanics as RPG Dreamer, would the proposed framework still achieve substantial improvements? Q4. What motivates the decision to measure playability primarily through quest completion, state consistency, and operational correctness? Is there additional evidence of gameplay depth, difficulty balance, or player experience?
4 comments
W3 & Q4: Scope and Motivation of the Playability Metric
We thank the reviewer for the comment on our playability metric. Rather than measuring subjective player enjoyment, our playability metric primarily evaluates whether a player can completely experience the created long-horizon game content, including progressing through quests across all branches, using all designed game functions, and maintaining reasonable behavior under edge-case interactions. We view this as the fundamental prerequisite for a game to be playable, whereas higher-level aspects of the gameplay experience, such as gameplay depth, difficulty balance, and overall engagement, are highly subjective and require more complex, dedicated investigation, which we discuss as future work. We have updated Section 4.2 in the revised manuscript to clarify the scope of the Playability metric.
If you have any further questions, we would be more than happy to continue the discussion.
I thank the authors for making the effort to address my questions. However, the paper will be much stronger if the authors can include more in-depth playability metric instead of leaving it as future work.
To the best of our knowledge, this is the first work to build a framework for long-horizon agentic game generation. We have put substantial effort into establishing this new direction:
- Decoupled Game Generation Framework: We design AOGE to decouple reusable game mechanics from game-specific content while providing unified observation and action interfaces for agents.
- Closed-Loop Generation Pipeline: We introduce RGL to iteratively evolve game content and target latent bugs via multi-reviewer playtesting.
- Long-Horizon RPG Dataset: We construct RPG-Bench, comprising 150 evaluation instances for generating hour-scale RPGs across diverse genres.
- Trajectory-Based Evaluation: We introduce an evaluation protocol grounded in dynamic gameplay trajectories across multiple modes, where playability specifically evaluates long-horizon stability.
We fully agree that a more in-depth playability metric would be valuable. We would like to respectfully clarify that our decision to leave this for future work is based on the following two considerations:
- Scope and Page Limit: Our work already encompasses a broad scope covering both the generation framework and the benchmark (as Reviewer WKF2 also noted). Adding extra metrics would make it difficult to explain everything clearly within the page limit.
- Time and Resource Constraints: Assessing higher-level aspects of the gameplay experience requires extensive subjective human evaluations. Conducting such evaluations reliably involves substantial time and annotation costs, making it infeasible to complete in the short term.
Reviewer fZ52
W1 & Q1: Causes of Uncompleted Routes
We thank the reviewer for raising this question regarding the failure causes in our complexity-scaling study. As described in Section 2.2 and Appendix D.1.1, for every generated game we extract its complete branching route structure from the source code in advance to serve as a walkthrough guide for the evaluators, thereby eliminating the influence of gameplay difficulty and insufficient in-game guidance. In fact, we inspected the failure causes of all uncompleted routes and found that every failure stems from implementation bugs, including unreasonable HP and combat stat configurations, erroneous dialogue branch conditions, and unreachable areas. Although some battles are challenging and require proper skill usage to win, evaluators can pass them by following the walkthrough guide, so combat difficulty is not an issue. We clarify this finding in the revised Appendix D.1.1.
W2 & Q2: Scope of Task and Constructed Mechanics
We thank the reviewer for this thoughtful question regarding the role of AOGE and the scope of our task.
Scope of the task. Our claim about generating complete, complex, long-horizon games refers to the fact that, compared with prior methods that produce only isolated code snippets, single levels, or text-only prototypes, our generated games are complete, self-contained games encompassing images, sprite animations, audio, and multi-chapter narratives. Meanwhile, rather than focusing on having agents invent intricate or novel game mechanics, our work focuses on reliably generating playable long-horizon games. Other systems for end-to-end game generation also use templates: OpenGame builds on project templates, while AutoUE builds on Unreal Engine gameplay modules and code templates.
Scope of the constructed mechanics. In our experimental setup, manually implemented means that we defined and fixed the required set of foundational RPG mechanics, prompted an agent to implement these requirements, manually verified that all required mechanics were properly implemented, and then kept these mechanics fixed when generating games across different requirements. Because games with different mechanics require different evaluation perspectives, we fixed the mechanics in our experiments to establish consistent evaluation rubrics.
Role of mechanics. While the constructed mechanics determine how a player interacts with the game, they do not guarantee playability. In fact, especially in games, many bugs arise from content rather than mechanics. For example, the mechanics define how objects, collision boxes, and interactive zones operate on a map, whereas the content specifies their exact positions and sizes; in practice, the bugs players frequently encounter are not game crashes upon entering a map, but offset positions or incorrect sizes that prevent players from obtaining required items to advance the progression. AOGE only specifies the overall architecture without restricting the game genre: users can either modify our provided one to add custom mechanics or prompt an agent to build a new version for other genres, such as visual novels or platformers.
We have revised Section 3 in the updated manuscript to clarify the construction of AOGE and the respective roles of mechanics and content.
W2 & Q3: Fair Comparison with Prebuilt Mechanics
We thank the reviewer for raising this concern regarding comparison fairness. To test whether RPG Dreamer's gains come merely from prebuilt game mechanics, we have added an ablation in Appendix B.5 (Table 8) evaluating multiple methods when provided with the exact same prebuilt mechanics used by RPG Dreamer:
As shown in the table, providing prebuilt mechanics alone brings only minor overall gains to the baselines (+4.0% for Kimi-code and +7.9% for Codex), whereas RPG Dreamer still outperforms Codex by 18.6% overall and 28.9% in playability under the same model and prebuilt mechanics. This confirms that prebuilt mechanics alone do not substantially improve baseline agents on long-horizon games without our agentic interface and reviewer-centric loop.