- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper presents GameWAM, a World Action Model for native closed-loop gameplay and GUI control. The framework jointly models future visual observations and keyboard/mouse actions using Video- and Action-DiTs. Experiments on bench including Minecraft and ViZDoom show competitive task performance with fewer executed actions. The paper also identifies LASI (Low-Frequency Action Source Imprinting), a failure mode in generative action prediction .
Critical strengths
- Application of WAMs to game environments seems to be interesting.
- Closed-loop results are promising.
- The investigation of LASI is a useful observation.
Critical weaknesses
1. Limited methodological novelty over existing WAMs. Several key components (asymmetric world-action prediction, modality-decoupled formulation, hierarchical history.. ) appear closely related to existing approaches. Although the paper demonstrates the effectiveness of each choice, it remains unclear what the main methodological novelty is beyond adapting and combining existing techniques for game control.
2. Unclear distinction from prior game world-action models. WHAM already models gameplay videos and controller actions and supports action prediction from visual observations. The paper could better explain which challenges are specific to the methods' setting.
3. Insufficient evaluation from a world-modeling perspective. The paper provides some analysis of video-side optimization, but lacks a direct quantitative evaluation of future visual prediction and action-observation consistency.
Questions for authors
I would appreciate the authors' responses to the concerns raised in the Weaknesses section.
4 comments
Response to Reviewer 9hEk (Part 2/2)
W2. Distinction from WHAM and prior game world models
We thank the reviewer for highlighting WHAM (Kanervisto et al., 2025). We agree that WHAM jointly models gameplay visuals and human controller actions, including action prediction from visual observations. However, the two models differ fundamentally in their inputs, learning objectives, and intended outputs. WHAM is primarily a generative model of gameplay: given previous game frames and controller actions from Bleeding Edge, it can predict future visual frames conditioned on specified controller inputs, or generate plausible player actions from visual history. It conditions on game visuals and controller inputs rather than natural-language task instructions. For example, WHAM can predict how a game scene evolves when a player moves the joystick, or what action a player might take next. GameWAM instead takes game observations and an explicit task instruction, such as "harvest sugar cane" in Minecraft, and generates executable keyboard–mouse actions to achieve that goal. It executes them in the actual game and repeatedly uses new observations to guide subsequent actions. Thus, WHAM models what may happen next in a game, whereas GameWAM learns what actions to take to complete a specified task.
Their overall modeling approaches also differ. WHAM uses VQGAN to encode gameplay frames into discrete tokens and an autoregressive Transformer trained through next-token prediction over visual and controller-action sequences. GameWAM instead follows the emerging generative World Action Model paradigm exemplified by DreamZero and Fast-WAM, jointly learning future visual observations and task-directed actions through Video- and Action-DiTs with flow matching. Accordingly, WHAM primarily evaluates the consistency, diversity, and persistence of generated gameplay sequences, while GameWAM evaluates actual closed-loop gameplay and GUI control in Minecraft and ViZDoom. GameWAM therefore addresses a different problem from WHAM: learning an instruction-conditioned control policy rather than generating plausible gameplay and player behavior. We will clarify this distinction in the revised manuscript.
W3. Quantitative world-modeling evaluation and action–observation consistency
We thank the reviewer for raising this important question and appreciate the opportunity to clarify the role of world modeling in GameWAM. We respectfully believe this concern partly reflects a misunderstanding of the objective of an action-oriented WAM versus a standalone world model. GameWAM is fundamentally an action model whose goal is to learn an executable policy for closed-loop game control. World modeling provides predictive supervision for this policy. Recent work further shows that effective world–action control does not require explicit future-video generation at inference. Fast-WAM skips test-time video prediction, ForeWAM uses latent future representations without video decoding, and DELE-w0.5 learns compact future-state latents instead of videos. Pixel-level video quality is thus neither a universally applicable nor a sufficient measure of action-relevant world modeling. Likewise, we are unaware of a standardized WAM metric for action–observation consistency. An inverse-dynamics model could infer actions from predicted observations, but distinct actions can produce visually indistinguishable transitions. A mismatch could therefore reflect ambiguity or evaluator error even when the predicted action and observation are consistent. Closed-loop execution directly tests whether the policy’s actions accomplish the instructed task. This is why WAM studies such as DreamZero and Fast-WAM place primary weight on closed-loop action performance. Following this practice, we evaluate how effectively GameWAM’s actions accomplish the instructed tasks in closed-loop interaction. We report task success and steps to completion on MCU and mean episode reward on ViZDoom. Like Fast-WAM, GameWAM generates no future frames during online rollout, so we cannot assess their quality against subsequent observations.
Nevertheless, to further address the reviewer’s concern, we evaluated future-video prediction offline on 350 samples by comparing predicted frames with the recorded future frames, obtaining 29.50 dB PSNR and 0.7726 SSIM. These metrics quantify visual reconstruction fidelity, but pixel-level similarity is sensitive to small viewpoint changes and does not directly establish whether the predicted observations agree with the predicted actions. As complementary evidence, our controlled action-only ablation shows that video supervision raises MCU Mini ASR from 35.7% to 50.7% (+15.0 percentage points, task-paired 95% CI [8.3, 21.7]), with gains of +10.0, +9.0, and +26.0 points for Embodied, GUI, and Combat tasks, respectively. Thus, we provide the requested video diagnostics while evaluating the practical benefit of auxiliary world modeling through the closed-loop control objective.
I appreciate the authors' efforts in addressing my concerns and the additional experiments provided in the rebuttal.
One concern that remains is the evaluation of action–observation consistency. The future-video prediction results and the video-supervision ablation are useful additions, but they do not directly show whether the predicted visual changes are consistent with the corresponding actions. An evaluation under controlled conditions would provide stronger evidence for this aspect of the world model.
The clarification regarding WHAM is also helpful. I now better understand the differences in their objectives and intended use. However, I would still like to see more evidence of what the task-conditioned, closed-loop formulation contributes beyond existing world-action modeling approaches.
That said, the rebuttal has addressed a substantial part of my initial concerns, and I appreciate the authors' effort. I am therefore raising my rating.
We sincerely thank the reviewer for their constructive follow-up and for reconsidering their rating.
We agree that action–observation consistency could be a useful direction for further investigation, while emphasizing that it would remain a supplementary metric rather than a substitute for closed-loop control performance. As discussed in our rebuttal, reliable evaluation is nontrivial because different actions can produce visually indistinguishable transitions, making it difficult to distinguish genuine inconsistencies from inherent ambiguity. Our PSNR/SSIM results and controlled ablations therefore provide complementary evidence without claiming to directly measure consistency.
Regarding methodological contributions, our matched-data Fast-WAM comparison (46.6% vs. 25.2% MCU All ASR) and component ablations support the value of our integrated task-conditioned, closed-loop formulation.
We appreciate the reviewer's thoughtful feedback and consideration.
Response to Reviewer 9hEk (Part 1/2)
We thank the reviewer for recognizing the promise of GameWAM's closed-loop results, the relevance of applying World Action Models to game environments, and the value of our LASI investigation. We appreciate the constructive feedback and address the three concerns below.
W1. Methodological novelty
We thank the reviewer for recognizing the effectiveness of our design choices and agree that our main contribution could be clearer. GameWAM naturally builds on modality-decoupled world–action modeling, with joint video–action supervision during training and action-only inference. We respectfully disagree, however, that adapting and combining existing techniques fully describes our contribution. Even when individual operations are familiar, we believe novelty can lie in the overall formulation, the challenges it resolves, and the scientific insights it yields. GameWAM contributes on all three fronts.
(1) Establishing WAMs as a competitive paradigm for native game control is itself an important insight. Unlike prevailing VLM-based reasoning agents, VLA-style policies, and heavily post-trained game agents, GameWAM shows that a task-conditioned WAM can achieve strong performance and interaction efficiency in native keyboard–mouse gameplay and GUI control. On MCU All, it achieves 46.6% ASR vs. 42.5% for Game-TARS, with substantially fewer executed actions per successful episode and no large-scale multi-game policy pretraining. Moreover, as a low-level executable policy, it can complement higher-level planners for strategy and task decomposition, offering a path to more capable long-horizon agents.
(2) Block–cycle control connects what the policy predicts, what it executes, and what it retains for the next cycle. Existing WAMs do not jointly address dense first-person interaction, frequent feedback, mixed native controls, and persistent context within a block-autoregressive policy. During training, overlapping, execution-aligned targets extend predictive supervision beyond the actions executed per block. During control, the policy executes a short action block, observes the outcome, and replans. Only realized observations enter its visual history. Hierarchical history combines fine-grained cycle-local KV states, compressed recent visuals, and gated long-term memory within a bounded token budget. We acknowledge that per-step gameplay/GUI routing and mode-specific action distributions are relatively straightforward engineering choices. Nevertheless, we identified a limitation of unified action modeling across interaction modes and demonstrated a simple, effective solution. MCU Mini ASR is 50.7% for GameWAM, versus 38.3% with unified action distributions, 41.3% with matched prediction–execution horizons, 38.0% without event-anchored sampling, and 36.7% with coarser temporal sampling. Removing cross-cycle history yields 46.7% (with a 12-point GUI drop), and removing future-video supervision yields 35.7%. Under matched Minecraft training data and evaluation, GameWAM achieves 46.6% MCU All ASR vs. 25.2% for reproduced Fast-WAM (+21.4 points), supporting the value of the complete formulation.
(3) Game control yields insights relevant beyond games. Reproducible, instrumented games expose challenges also found in dynamic, cluttered, partially observable physical environments, including rapid visual changes, heterogeneous actions, frequent feedback, persistent world state, and limited temporal context. GameWAM offers concrete principles for conditioning actions on interaction semantics, separating predictive look-ahead from execution commitment, and retaining structured context across replanning cycles. These principles offer methodological guidance from controlled game environments and motivate further exploration of embodied WAMs in physical settings.
(4) LASI is an independent scientific discovery in generative control. We identify frequency-specific causal imprinting of stochastic action sources on coarse camera motion, beyond ordinary noise dependence. For yaw DCT0, low-frequency donor swaps yield 94.8% donor-following, and zeroing the same band removes 99.25% of its source-induced variance. Antithetic and closed-loop permutation tests provide further evidence. LASI is independent of source reuse, which amplifies directional bias into persistent rotation and task failures. Additional fixed-source experiments reduce MCU Mini ASR from 50.7% to 13.7% and MCU All from 46.6% to 19.3%, demonstrating practical harm. More broadly, LASI highlights a potentially general mechanism by which stochastic generation sources influence action behavior, opening new directions for analyzing and improving generative control beyond games.
Together, these contributions establish GameWAM as a substantive methodological and scientific advance beyond a mere engineering adaptation.