acceptodds
Under review as a conference paper at ICLR 2027

GameDirector: Decoupling Gameplay Logic from Rendering for Player-Configurable Game World Models

21 views

Abstract

Recent game world models support realistic visual simulation and interactive gameplay based on player inputs. However, they typically learn environment dynamics from pixel-level supervision, jointly modeling perception, memory, state transitions, and rendering within an end-to-end framework. While this design enables open-ended, action-controllable generation, it still falls short of delivering a complete gameplay experience. Games are governed by explicit mechanics, such as health deduction, skill activation, combat rules, and termination conditions. These mechanics depend on precise and consistent state transitions that generative models alone cannot reliably enforce. In contrast, game engines can guarantee such mechanics through hard-coded rules, but provide limited flexibility for player-driven creation. To bridge these paradigms, we introduce GameDirector, the first agentic framework that decouples rule-based gameplay logic from visual rendering. Given player-defined configurations, the framework acts as an intelligent director that interprets visual observations, updates game states, tactically controls NPCs, and enforces gameplay rules. It then translates these decisions into text prompts that guide the video world model to render the resulting gameplay. This separation allows players to configure characters, states, and rules much like a game developer while preserving coherent game mechanics. Experiments on three games, using data collected by our automated gameplay agent, show that GameDirector achieves accurate state tracking, reliable rule following, and improves boss action quality by more than 39.9% over various end-to-end game world model settings. Overall, by externalizing player-controllable game logic, GameDirector establishes an effective middle ground between hard-coded simulation and generative modeling, enabling more flexible and closed-loop gameplay experiences.

Primary area:
generative models
Keywords:
Agentic Framework, Game World Model

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_hnTDOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper presents GameDirector, a framework that decouples gameplay logic from video generation. It introduces an agentic control layer for state tracking, action planning, and rule enforcement to generate gameplay videos that follow player-configurable rules.

Critical strengths
  • The authors build a complete pipeline for generating gameplay videos with consistent game logic and achieve promising results.
  • The paper is well-organized, clearly written, and easy to follow.
Critical weaknesses
  • The methodological novelty is limited. The authors have already discussed several similar planner-video generator frameworks in the related work. I suggest that the authors identify the most significant difference between GameDirector and existing methods. "Jointly integrate" multiple components is not a convincing answer in terms of novelty.
  • The motivation for jointly training AttackNet and SituationNet across three games is unclear. Why do the authors train shared models on all three games instead of training a separate model for each game? The games shown in the paper have substantially different visual appearances and styles. It would be helpful to explain the benefits of joint training or provide a comparison with game-specific models.
  • The State Alignment metric is not convincing. GameDirector uses AttackNet predictions to update game states and guide subsequent frame generation, while the evaluation also relies on AttackNet to determine hit events. Therefore, achieving 100% State Alignment is expected by design. This result mainly demonstrates the limitations of previous approaches in maintaining deterministic state consistency. I have similar concerns about the Mechanics Fidelity and Decision Quality metrics, making the evaluation somewhat tailored to support the desired conclusions. I suggest that the authors reconsider these evaluations or at least moderate the corresponding claims.
Questions for authors
  • The current system appears closer to a proof-of-concept prototype, as it relies on game-specific training and the maintenance of predefined skills and rules. This may limit its potential for scaling to more diverse and complex games. How do the authors envision extending the framework to a broader range of games without substantial game-specific effort?
  • How does the framework perform over much longer gameplay durations? I am curious whether the model suffers from drifting or accumulated visual inconsistencies during long rollouts. For example, if both characters are configured with very high HP and very low attack damage Would the framework remain stable and visually consistent in such scenarios?
Reviewer_X7g9OpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

Authors argue that jointly modelling game mechanics and game visuals in with world models fails to produce game-like experiences. They seek to address this by decoupling the two with an agentic framework handing the mechanics (game state, rules, memory, etc.) and a generative model handling the visuals. They believe the benefit of generative video game world models is the open ended visual generation, so they seek to preserve this feature while pushing the mechanics towards an agentic framework, developing baselines for those mechanics and comparing the agent's performance against an end-to-end world model baseline.

Critical strengths

The trend has certainly been toward deep world model architectures that focus on video and actions and assume the game mechanics emerge from scale. The authors point out that what makes a game is more than the visuals or event the "physics" underlying the visuals, but the systems that make gameplay compelling to a gamer.

If having game mechanics be implicit in a trainable world model architecture is one approach. The opposite end of the spectrum is using a conventional game engine to handle the game logic, which of course is near equivalent to coding a game from scratch. The agentic pipeline prposed here provides a nice compromise -- it makes specific elements essential to gameplay (combat parameters, health bars, skill eligibility, cooldowns, death) explicit and validatable. The combination with visual feedback differentiates the model from a more conventional game engine. I'm not sure if it is the right approach for every use case, but it is a useful read for those trying to understand how to integrate world models into video games.

Critical weaknesses

Would have been nice to evaluate game mechanic fidelity and action quality over closed-loop video rollouts instead of just held out data.

Questions for authors

See above comment under weakness

Reviewer_kQHdOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

Authors propose GameDirector, which separates gameplay logic from video rendering. Perception modules estimate hits and spatial relations, explicit rules maintain health and skill state, and a language model selects boss actions to guide the video generator. This design allows users to change combat settings without retraining the generator. The authors collect annotated clips from three games and compare several control designs using the same video backbone. The results suggest improved gameplay consistency and action selection.

Critical strengths
  1. The paper presents a practical and reasonably complete system. The state updates, skill rules, and prompt interface make it clear how users can change health, damage, cooldowns, and other combat settings. The configurable-combat and cross-character skill examples are interesting demonstrations of this design.

  2. The comparisons share the video backbone, training data, player controls, and evaluation samples, which helps assess the different control designs. The authors also provide useful perception and runtime results. In particular, the runtime breakdown shows that asynchronous planning adds relatively little overhead to video generation.

Critical weaknesses

1.The headline scores do not independently establish correctness in the generated video. State Alignment uses the same AttackNet outputs that update the state, so the reported 100% mainly reflects consistent bookkeeping, as the authors acknowledge. AttackNet's accuracy on held-out game footage does not tell us its accuracy on generated videos or how its mistakes accumulate into health errors. Decision Quality is also judged from SituationNet's predicted geometry without showing the video. If that geometry is wrong, an action can receive a favorable score while being unsuitable for the actual scene. An independent evaluation of the generated rollouts would make these results much more convincing.

Questions for authors
  1. How accurate are hit detection and spatial estimates on independently annotated generated videos? What is the resulting cumulative health error over a rollout, and how often does a high State Alignment score coincide with an incorrect visible outcome?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.