acceptodds
Under review as a conference paper at ICLR 2027

GameWAM: A World Action Model for Video Games

Abstract

Modern video games combine first-person perception, rapid visual changes, persistent world state, and heterogeneous native controls. Existing game agents map visual and task context directly to actions but lack explicit world dynamics modeling, whereas interactive game world models predict visual futures from supplied actions but do not serve as task policies. World Action Models (WAMs) unify these objectives, but remain largely unexplored under the dynamics and open-ended interaction of video games. We introduce GameWAM, to our knowledge the first WAM for native closed-loop gameplay and GUI control. GameWAM jointly generates future visual observations and executable keyboard-mouse trajectories through parallel visual and action generative processes with block-causal conditioning and flow matching. To support joint world-action learning, we construct synchronized gameplay and GUI trajectories. To handle heterogeneous native control, GameWAM predicts a gameplay/GUI mode per action step and generates actions with mode-specific prediction distributions and continuous-action normalization. For long-horizon interaction, block-cycle control coordinates prediction, execution, and temporal context: it predicts beyond the committed horizon, executes short action blocks, replans from new observations, and hierarchically structures context from fine-grained within-cycle history to persistent cross-cycle history. Experiments show competitive task success with fewer executed native actions than the compared agents. We further uncover Low-Frequency Action Source Imprinting (LASI), in which low-frequency components of the sampled action source systematically steer coarse generated camera motion under fixed conditioning, revealing a source-sensitivity failure mode in generative control.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
video game agents, world action models, closed-loop control, flow matching

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_9hEkOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper presents GameWAM, a World Action Model for native closed-loop gameplay and GUI control. The framework jointly models future visual observations and keyboard/mouse actions using Video- and Action-DiTs. Experiments on bench including Minecraft and ViZDoom show competitive task performance with fewer executed actions. The paper also identifies LASI (Low-Frequency Action Source Imprinting), a failure mode in generative action prediction .

Critical strengths
  1. Application of WAMs to game environments seems to be interesting.
  2. Closed-loop results are promising.
  3. The investigation of LASI is a useful observation.
Critical weaknesses

1. Limited methodological novelty over existing WAMs. Several key components (asymmetric world-action prediction, modality-decoupled formulation, hierarchical history.. ) appear closely related to existing approaches. Although the paper demonstrates the effectiveness of each choice, it remains unclear what the main methodological novelty is beyond adapting and combining existing techniques for game control.

2. Unclear distinction from prior game world-action models. WHAM already models gameplay videos and controller actions and supports action prediction from visual observations. The paper could better explain which challenges are specific to the methods' setting.

3. Insufficient evaluation from a world-modeling perspective. The paper provides some analysis of video-side optimization, but lacks a direct quantitative evaluation of future visual prediction and action-observation consistency.

Questions for authors

I would appreciate the authors' responses to the concerns raised in the Weaknesses section.

4 comments
AuthorsOpenReview ↗

Response to Reviewer 9hEk (Part 1/2)

We thank the reviewer for recognizing the promise of GameWAM's closed-loop results, the relevance of applying World Action Models to game environments, and the value of our LASI investigation. We appreciate the constructive feedback and address the three concerns below.

W1. Methodological novelty

We thank the reviewer for recognizing the effectiveness of our design choices and agree that our main contribution could be clearer. GameWAM naturally builds on modality-decoupled world–action modeling, with joint video–action supervision during training and action-only inference. We respectfully disagree, however, that adapting and combining existing techniques fully describes our contribution. Even when individual operations are familiar, we believe novelty can lie in the overall formulation, the challenges it resolves, and the scientific insights it yields. GameWAM contributes on all three fronts.

(1) Establishing WAMs as a competitive paradigm for native game control is itself an important insight. Unlike prevailing VLM-based reasoning agents, VLA-style policies, and heavily post-trained game agents, GameWAM shows that a task-conditioned WAM can achieve strong performance and interaction efficiency in native keyboard–mouse gameplay and GUI control. On MCU All, it achieves 46.6% ASR vs. 42.5% for Game-TARS, with substantially fewer executed actions per successful episode and no large-scale multi-game policy pretraining. Moreover, as a low-level executable policy, it can complement higher-level planners for strategy and task decomposition, offering a path to more capable long-horizon agents.

(2) Block–cycle control connects what the policy predicts, what it executes, and what it retains for the next cycle. Existing WAMs do not jointly address dense first-person interaction, frequent feedback, mixed native controls, and persistent context within a block-autoregressive policy. During training, overlapping, execution-aligned targets extend predictive supervision beyond the actions executed per block. During control, the policy executes a short action block, observes the outcome, and replans. Only realized observations enter its visual history. Hierarchical history combines fine-grained cycle-local KV states, compressed recent visuals, and gated long-term memory within a bounded token budget. We acknowledge that per-step gameplay/GUI routing and mode-specific action distributions are relatively straightforward engineering choices. Nevertheless, we identified a limitation of unified action modeling across interaction modes and demonstrated a simple, effective solution. MCU Mini ASR is 50.7% for GameWAM, versus 38.3% with unified action distributions, 41.3% with matched prediction–execution horizons, 38.0% without event-anchored sampling, and 36.7% with coarser temporal sampling. Removing cross-cycle history yields 46.7% (with a 12-point GUI drop), and removing future-video supervision yields 35.7%. Under matched Minecraft training data and evaluation, GameWAM achieves 46.6% MCU All ASR vs. 25.2% for reproduced Fast-WAM (+21.4 points), supporting the value of the complete formulation.

(3) Game control yields insights relevant beyond games. Reproducible, instrumented games expose challenges also found in dynamic, cluttered, partially observable physical environments, including rapid visual changes, heterogeneous actions, frequent feedback, persistent world state, and limited temporal context. GameWAM offers concrete principles for conditioning actions on interaction semantics, separating predictive look-ahead from execution commitment, and retaining structured context across replanning cycles. These principles offer methodological guidance from controlled game environments and motivate further exploration of embodied WAMs in physical settings.

(4) LASI is an independent scientific discovery in generative control. We identify frequency-specific causal imprinting of stochastic action sources on coarse camera motion, beyond ordinary noise dependence. For yaw DCT0, low-frequency donor swaps yield 94.8% donor-following, and zeroing the same band removes 99.25% of its source-induced variance. Antithetic and closed-loop permutation tests provide further evidence. LASI is independent of source reuse, which amplifies directional bias into persistent rotation and task failures. Additional fixed-source experiments reduce MCU Mini ASR from 50.7% to 13.7% and MCU All from 46.6% to 19.3%, demonstrating practical harm. More broadly, LASI highlights a potentially general mechanism by which stochastic generation sources influence action behavior, opening new directions for analyzing and improving generative control beyond games.

Together, these contributions establish GameWAM as a substantive methodological and scientific advance beyond a mere engineering adaptation.

AuthorsOpenReview ↗

Response to Reviewer 9hEk (Part 2/2)

W2. Distinction from WHAM and prior game world models

We thank the reviewer for highlighting WHAM (Kanervisto et al., 2025). We agree that WHAM jointly models gameplay visuals and human controller actions, including action prediction from visual observations. However, the two models differ fundamentally in their inputs, learning objectives, and intended outputs. WHAM is primarily a generative model of gameplay: given previous game frames and controller actions from Bleeding Edge, it can predict future visual frames conditioned on specified controller inputs, or generate plausible player actions from visual history. It conditions on game visuals and controller inputs rather than natural-language task instructions. For example, WHAM can predict how a game scene evolves when a player moves the joystick, or what action a player might take next. GameWAM instead takes game observations and an explicit task instruction, such as "harvest sugar cane" in Minecraft, and generates executable keyboard–mouse actions to achieve that goal. It executes them in the actual game and repeatedly uses new observations to guide subsequent actions. Thus, WHAM models what may happen next in a game, whereas GameWAM learns what actions to take to complete a specified task.

Their overall modeling approaches also differ. WHAM uses VQGAN to encode gameplay frames into discrete tokens and an autoregressive Transformer trained through next-token prediction over visual and controller-action sequences. GameWAM instead follows the emerging generative World Action Model paradigm exemplified by DreamZero and Fast-WAM, jointly learning future visual observations and task-directed actions through Video- and Action-DiTs with flow matching. Accordingly, WHAM primarily evaluates the consistency, diversity, and persistence of generated gameplay sequences, while GameWAM evaluates actual closed-loop gameplay and GUI control in Minecraft and ViZDoom. GameWAM therefore addresses a different problem from WHAM: learning an instruction-conditioned control policy rather than generating plausible gameplay and player behavior. We will clarify this distinction in the revised manuscript.

W3. Quantitative world-modeling evaluation and action–observation consistency

We thank the reviewer for raising this important question and appreciate the opportunity to clarify the role of world modeling in GameWAM. We respectfully believe this concern partly reflects a misunderstanding of the objective of an action-oriented WAM versus a standalone world model. GameWAM is fundamentally an action model whose goal is to learn an executable policy for closed-loop game control. World modeling provides predictive supervision for this policy. Recent work further shows that effective world–action control does not require explicit future-video generation at inference. Fast-WAM skips test-time video prediction, ForeWAM uses latent future representations without video decoding, and DELE-w0.5 learns compact future-state latents instead of videos. Pixel-level video quality is thus neither a universally applicable nor a sufficient measure of action-relevant world modeling. Likewise, we are unaware of a standardized WAM metric for action–observation consistency. An inverse-dynamics model could infer actions from predicted observations, but distinct actions can produce visually indistinguishable transitions. A mismatch could therefore reflect ambiguity or evaluator error even when the predicted action and observation are consistent. Closed-loop execution directly tests whether the policy’s actions accomplish the instructed task. This is why WAM studies such as DreamZero and Fast-WAM place primary weight on closed-loop action performance. Following this practice, we evaluate how effectively GameWAM’s actions accomplish the instructed tasks in closed-loop interaction. We report task success and steps to completion on MCU and mean episode reward on ViZDoom. Like Fast-WAM, GameWAM generates no future frames during online rollout, so we cannot assess their quality against subsequent observations.

Nevertheless, to further address the reviewer’s concern, we evaluated future-video prediction offline on 350 samples by comparing predicted frames with the recorded future frames, obtaining 29.50 dB PSNR and 0.7726 SSIM. These metrics quantify visual reconstruction fidelity, but pixel-level similarity is sensitive to small viewpoint changes and does not directly establish whether the predicted observations agree with the predicted actions. As complementary evidence, our controlled action-only ablation shows that video supervision raises MCU Mini ASR from 35.7% to 50.7% (+15.0 percentage points, task-paired 95% CI [8.3, 21.7]), with gains of +10.0, +9.0, and +26.0 points for Embodied, GUI, and Combat tasks, respectively. Thus, we provide the requested video diagnostics while evaluating the practical benefit of auxiliary world modeling through the closed-loop control objective.

I appreciate the authors' efforts in addressing my concerns and the additional experiments provided in the rebuttal.

One concern that remains is the evaluation of action–observation consistency. The future-video prediction results and the video-supervision ablation are useful additions, but they do not directly show whether the predicted visual changes are consistent with the corresponding actions. An evaluation under controlled conditions would provide stronger evidence for this aspect of the world model.

The clarification regarding WHAM is also helpful. I now better understand the differences in their objectives and intended use. However, I would still like to see more evidence of what the task-conditioned, closed-loop formulation contributes beyond existing world-action modeling approaches.

That said, the rebuttal has addressed a substantial part of my initial concerns, and I appreciate the authors' effort. I am therefore raising my rating.

We sincerely thank the reviewer for their constructive follow-up and for reconsidering their rating.

We agree that action–observation consistency could be a useful direction for further investigation, while emphasizing that it would remain a supplementary metric rather than a substitute for closed-loop control performance. As discussed in our rebuttal, reliable evaluation is nontrivial because different actions can produce visually indistinguishable transitions, making it difficult to distinguish genuine inconsistencies from inherent ambiguity. Our PSNR/SSIM results and controlled ablations therefore provide complementary evidence without claiming to directly measure consistency.

Regarding methodological contributions, our matched-data Fast-WAM comparison (46.6% vs. 25.2% MCU All ASR) and component ablations support the value of our integrated task-conditioned, closed-loop formulation.

We appreciate the reviewer's thoughtful feedback and consideration.

Reviewer_ob9DOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

GameWAM introduces a World Action Model for native gameplay and GUI interaction, jointly learning future video prediction and action generation from synchronized visual observations and keyboard–mouse trajectories. It uses mode-aware action branches to handle gameplay and interface operations, alongside a predict-long, execute-short strategy and hierarchical visual history to adapt actions to new observations during continuous interaction. Video prediction remains a training objective, while default online inference generates actions without the additional cost of future-video generation. Experiments on Minecraft and ViZDoom demonstrate competitive closed-loop performance, with fewer executed actions in successful Minecraft episodes. The paper also identifies Low-Frequency Action Source Imprinting (LASI), showing how low-frequency components of the generative noise influence camera motion and how repeatedly reusing the same noise can accumulate into persistent directional bias.

Critical strengths

The method addresses practical challenges in game control. It distinguishes gameplay and GUI action distributions and maintains timely feedback through a predict-long, execute-short strategy. Ablation studies support the benefits of these designs for control performance.

The LASI analysis has substantial experimental support. Low-frequency noise replacement and zeroing under fixed conditioning, together with closed-loop trajectory analysis, support a causal influence of source noise on camera motion rather than mere correlation.

Critical weaknesses

The independent benefit of hierarchical history remains unclear. The existing ablation primarily compares retaining versus removing cross-cycle history. While it supports the usefulness of historical information, it does not distinguish the benefit of additional history from that of the hierarchical memory organization itself.

The practical control impact of LASI is insufficiently quantified. The experiments convincingly demonstrate that low-frequency source noise affects camera motion, but it remains unclear how many task failures this causes and how much task performance resampling recovers.

Questions for authors

What additional value does hierarchical history provide? Does it outperform a simpler history-retention scheme under the same context budget? Do the gains primarily come from additional historical information or from the hierarchical organization and compression mechanisms?

How much does LASI affect actual task performance? Under the same tasks and initial conditions, how do task success rates and persistent rotation rates differ between fixed-source reuse and resampling at each replanning step? Can these results quantify LASI’s practical harm and the benefits of resampling?

1 comment
AuthorsOpenReview ↗

Response to Reviewer ob9D

We thank the reviewer for recognizing the practical value of GameWAM's mode-aware action modeling and predict-long, execute-short control strategy, the empirical support provided by our ablations, and the rigorous causal evidence underlying our LASI analysis. We appreciate the reviewer's constructive feedback and address the two concerns below.

W1 / Q1. Independent benefit of hierarchical history

We thank the reviewer for this constructive suggestion. To distinguish the benefit of hierarchical organization from simply retaining more historical information, we additionally evaluate a FIFO-only baseline under the same 40-token cross-cycle context budget. Both approaches use the same VAE-based visual representation and Conv3D tokenization. The FIFO baseline retains the 40 most recent compressed visual tokens without learned long-term aggregation. In contrast, GameWAM preserves 32 tokens from the two most recent completed cycles and allocates eight tokens to learned multi-timescale memory, summarizing older executed observations through attention-based updates and content-dependent gating (Appendix A.3 and C.2).

We compare these approaches with the existing no-history ablation on MCU Mini:

MethodEmbodiedGUICombatAvg.
Full GameWAM70.0 ± 32.243.0 ± 32.939.0 ± 25.950.7
Recent-only history (budget-matched)75.0 ± 32.333.0 ± 38.736.0 ± 30.748.0
No cross-cycle history75.0 ± 30.131.0 ± 29.834.0 ± 31.046.7

ASR (%) is equally averaged across the three task categories; ±SD denotes variation across per-task success rates.

Under the identical 40-token context budget, hierarchical history achieves 50.7% average ASR versus 48.0% for the recent-only baseline (+2.7 percentage points), with a particularly pronounced improvement on GUI tasks (+10.0 points) and an additional gain on Combat (+3.0 points), while Embodied performance is 5.0 points lower. Removing cross-cycle history entirely yields 46.7% ASR. With category-stratified, task-paired bootstrap resampling (200,000 resamples), the 95% confidence interval for the hierarchical-versus-recent-only improvement is [−3.3, 9.0] percentage points. The positive aggregate point estimate is therefore not statistically conclusive, but the comparison demonstrates a task-dependent advantage, particularly for GUI interactions requiring persistent context across control cycles. We will incorporate this new controlled comparison into the revised manuscript.

W2 / Q2. Practical control impact of LASI

We thank the reviewer for recognizing the causal evidence for LASI and for suggesting a direct evaluation of its practical impact. To further quantify this effect, we additionally compare fixed-source reuse with GameWAM's standard action-source resampling at every replanning step, using the same MCU task sets and evaluation protocol:

Action sourceEmbodied MiniEmbodied AllGUI MiniGUI AllCombat MiniCombat AllAvg. MiniAvg. All
Fixed-source reuse2.00.338.056.81.00.813.719.3
Resampling70.047.543.060.039.032.250.746.6

Fixed-source reuse reduces average ASR by 37.0 percentage points on MCU Mini and 27.3 points on MCU All. The effect is particularly severe in Embodied and Combat tasks, where successful control relies heavily on camera orientation, whereas GUI performance is much less affected. Regarding persistent rotation, as demonstrated in Section 5.5 and Appendix D.7.8, this phenomenon refers to a directional camera bias that persists across successive replanning steps under fixed-source reuse. Since camera turning is also a necessary part of normal gameplay, defining a binary rotation occurrence would require additional thresholds for angular displacement and persistence duration, which may conflate normal turning with pathological rotation. Moreover, camera rotation does not apply to GUI interactions, where the mouse controls an interface cursor. We therefore quantify the practical consequences through closed-loop task success rates, which directly measure whether the accumulated control bias prevents task completion. The pronounced performance degradation in gameplay tasks, together with the observed persistent rotation trajectories, demonstrates the practical harm of LASI and the effectiveness of source resampling. We will incorporate this additional comparison into the revised manuscript.

Reviewer_GoLVOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

GameWAM is a World Action Model for native keyboard–mouse control in Minecraft and ViZDoom. Parallel Video (Wan2.2-5B-initialized) and Action DiTs, with block-causal attention and joint flow matching, generate future frames and actions.

Key design elements are: a per-step gameplay/GUI router with mode-specific continuous-action normalization; predict-16/execute-8 block-cycle control; hierarchical memory: cycle KV cache, recent FIFO segments, and gated long-term slots.

On MCU, it reaches 50.7 (Mini) / 46.6 (All) ASR, above Game-TARS, with fewer steps per success. The paper also reports LASI: low-frequency components of the sampled noise steer generated camera motion, causing in-place rotation when one noise sample is reused across replanning steps.

Critical strengths

Strengths

  1. Extends WAMs from tabletop manipulation to fast-changing first-person games, covering both gameplay and GUI interaction, which is a meaningful setting.

  2. Per-step mode routing with mode-specific normalization is a sensible answer to the same keyboard–mouse channels carrying different semantics in gameplay and GUI, and the ablation shows a clear drop with a unified distribution.

  3. The LASI analysis is rigorous: donor swaps, zeroing, antithetic pairs, and a closed-loop permutation test give causal evidence that low-frequency noise components steer camera motion.

  4. Training cost is transparent (about 22 hours on 8 H200s), token usage is far below Game-TARS, and no checkpoint selection is used.

Critical weaknesses
  1. Limited novelty. The core components come from DreamZero, Fast-WAM, and Diffusion Policy; the paper is essentially an existing WAM transferred to games. The game-specific additions (mode routing, hierarchical memory) are largely engineering, and their necessity is not shown by controlled experiments.

  2. Uncontrolled comparisons. All baselines except Fast-WAM are copied from original reports. Training data and the MCU benchmark both derive from MineStudio, and GUI success is unusually high (60.0 vs. Game-TARS 39.1); an overlap analysis between training data and evaluation tasks is needed.

  3. On ViZDoom, the model is trained on expert data from the same maps but compared against zero-shot general models. The expert's own score and a behavior-cloning baseline on the same data are missing.

  4. Ablations are single runs with SD around ±30, so differences may not be significant. Steps are averaged over successful episodes only, introducing selection bias.

  5. LASI arises only under noise reuse, which is not standard practice, and no quantitative success-rate comparison of reused vs. resampled noise is reported.

Questions for authors
  1. How much do the event-anchored and scripted GUI data overlap with MCU evaluation tasks, especially crafting recipes?

  2. What is the APPO expert’s ViZDoom score, and how does a behavior-cloning baseline trained on the same data perform?

  3. Are steps counted at the same granularity for all methods? Can you report steps over all episodes, or success-vs-steps curves?

  4. What is the closed-loop ASR with a reused vs. resampled action source?

  5. Can the ablations be run on MCU All with multiple seeds and confidence intervals?

  6. How good is the future-video prediction? Beyond the action-only ablation, what evidence shows the world-modeling component helps closed-loop control?

4 comments
AuthorsOpenReview ↗

Response to Reviewer GoLV (Part 1/4)

We thank the reviewer for recognizing the significance of our game-control setting, the benefits of mode-specific modeling, the rigor of LASI analysis, and the transparency of our efficiency and evaluation protocols. We nevertheless believe some concerns stem from misunderstandings of our contributions or evaluation setting, which we clarify below.

W1. Methodological novelty

GameWAM is, by definition, a World Action Model. We naturally build on this paradigm without claiming to invent its foundational architecture. However, we respectfully disagree that it is merely an engineering adaptation. Novelty concerns the overall formulation, challenges addressed, and scientific insights, not just individual primitives. GameWAM contributes on all three fronts.

(1) Establishing WAMs as a competitive paradigm for native game control is itself an important insight. Unlike prevailing VLM-based reasoning agents, VLA-style policies, and heavily post-trained game agents, GameWAM shows that a task-conditioned WAM can achieve strong performance and interaction efficiency in native keyboard–mouse gameplay and GUI control. On MCU All, it achieves 46.6% ASR vs. 42.5% for Game-TARS, with substantially fewer executed actions per successful episode and no large-scale multi-game policy pretraining. Moreover, as a low-level executable policy rather than a complete reasoning agent, it can complement higher-level planners for strategy and task decomposition, offering a path to more capable long-horizon agents.

(2) Block–cycle control and hierarchical history are methodological designs, not arbitrary engineering combinations. Existing WAMs do not jointly address dense first-person interaction, frequent feedback, mixed native controls, and persistent context within a block-autoregressive policy. Our block–cycle AR coordinates training, prediction, execution, replanning, and memory: overlapping, execution-aligned targets extend supervision without longer action commitments; short execution blocks provide rapid feedback from realized observations; and unexecuted predictions never enter observed history. Hierarchical history combines fine-grained cycle-local KV states, compressed recent visuals, and gated long-term memory within a bounded token budget. We acknowledge that per-step gameplay/GUI routing and mode-specific action distributions are relatively straightforward engineering choices. Nevertheless, identifying the performance limits of unified modeling across interaction modes and proposing a simple, effective solution provides a practical design insight. MCU Mini ASR is 50.7% for GameWAM, versus 38.3% with unified action distributions, 41.3% with matched prediction–execution horizons, 38.0% without event-anchored sampling, and 36.7% with coarser temporal sampling. Removing cross-cycle history yields 46.7% (with a 12-point GUI drop); removing future-video supervision yields 35.7%. Notably, Strength 2 recognizes the benefits of mode-specific modeling, while Weakness 1 questions our additions; these ablations directly address that concern. Under matched Minecraft training data and evaluation, GameWAM achieves 46.6% MCU All ASR vs. 25.2% for reproduced Fast-WAM (+21.4 points), supporting our formulation beyond transferring an existing WAM.

(3) Game control yields insights relevant beyond games. Reproducible, instrumented games expose challenges shared with embodied intelligence in dynamic, cluttered, partially observable physical environments: rapid visual changes, heterogeneous actions, frequent feedback, persistent world state, and limited temporal context. GameWAM offers concrete principles: condition actions on interaction semantics, separate predictive look-ahead from execution commitment, and retain structured context across replanning cycles. These principles offer methodological guidance from controlled game environments and motivate further exploration of embodied WAMs in physical settings.

(4) LASI is an independent scientific discovery in generative control. We identify frequency-specific causal imprinting of stochastic action sources on coarse camera motion, beyond ordinary noise dependence. For yaw DCT0, low-frequency donor swaps yield 94.8% donor-following, and zeroing the same band removes 99.25% of its source-induced variance; antithetic and closed-loop permutation tests provide further evidence. LASI is independent of source reuse, which amplifies directional bias into persistent rotation and task failures. Additional fixed-source experiments reduce MCU Mini ASR from 50.7% to 13.7% and MCU All from 46.6% to 19.3%, demonstrating practical harm. More broadly, LASI highlights a potentially general mechanism by which stochastic generation sources influence action behavior, opening new directions for analyzing and improving generative control beyond games.

Together, these contributions establish GameWAM as a substantive methodological and scientific advance beyond a mere engineering adaptation.

AuthorsOpenReview ↗

Response to Reviewer GoLV (Part 2/4)

W2 / Q1. Training–evaluation overlap and benchmark fairness

We appreciate the request for clarification, but believe this concern conflates task-semantic overlap, evaluation-trajectory leakage, and controlled methodological comparison. As clarified in Appendix C.6, MCU evaluates the closed-loop execution of learned Minecraft skills, rather than zero-shot generalization to unseen recipes or task labels; zero-shot cross-game transfer is separately evaluated in Appendix D.6. Importantly, we follow the established MCU evaluation protocol introduced by OpenHA, which evaluates agents across procedurally varied worlds, seeds, and spawn positions. Even when task objectives are identical, different initial conditions and ongoing interactions produce diverse and unpredictable situations, requiring agents to adapt continuously to actual observations rather than replay memorized demonstrations. This is common in supervised learning: image classification shares class labels across training and test sets, while embodied agents are often trained and evaluated on the same task objectives. Supervised evaluation and zero-shot generalization to unseen tasks are fundamentally different evaluation settings. To further address the reviewer's concern and reaffirm our commitment to public dataset release, we additionally conduct an exact task-ID coverage analysis, finding 99.34% Embodied, 99.84% GUI, and 96.72% Combat coverage, or 99.51% overall. Such task-semantic overlap does not establish evaluation-trajectory leakage.

Importantly, both our Regular VPT and Event-Anchored VPT training streams originate from OpenAI's VPT gameplay recordings, rather than demonstrations collected specifically for MCU evaluation tasks. Regular VPT provides naturalistic gameplay trajectories, while Event-Anchored VPT extracts event-centered segments from the same recordings using Minecraft interaction statistics. Their semantic overlap with MCU naturally follows from shared Minecraft interaction definitions. We additionally construct scripted GUI trajectories to teach the crafting and inventory procedures necessary for native-action control (Appendix B.5). Such task-oriented supervision is well established in Minecraft research: OpenHA transforms VPT demonstrations into different action representations; ROCKET-1 relabels Contractor trajectories with interaction events and object segmentations; and JARVIS-VLA combines Contractor and agent-generated trajectories with synthesized crafting/smelting demonstrations. Notably, JARVIS-VLA explicitly reports collecting over 100 training trajectories for each evaluated MCU task, corresponding to 100% task-level training coverage of its evaluated task set. Crucially, GUI procedural supervision is essential for GameWAM as a low-level executable policy: learning to move the cursor or manipulate inventory slots does not teach which ingredients a recipe requires or how to arrange them. Without such demonstrations, GameWAM has no explicit recipe knowledge base or high-level planner from which to obtain this knowledge. In contrast, pretrained LLMs/VLMs may already encode crafting recipes, while other agents can access external instructions or knowledge resources. It would therefore be inconsistent to consider recipe knowledge learned through demonstrations inherently unfair while accepting the same knowledge acquired through pretraining or external resources. Regarding our 60.0% GUI ASR versus Game-TARS's 39.1%, the systems differ in training mixtures, representations, and targeted enhancements, while Game-TARS does not disclose its exact recipe-level training composition. We have never claimed that this performance gap is solely attributable to architectural differences.

More fundamentally, heterogeneous training limits causal attribution; it does not invalidate system-level benchmarking. As explicitly stated in Appendix C.6, Table 1 compares final agent capabilities under different training conditions rather than isolating architectural effects. Such comparisons are standard in foundation-model research; requiring identical training data would change the benchmark's purpose and exclude systems with unreleased training resources. Crucially, we also provide a controlled methodological comparison: reproduced Fast-WAM uses the same Minecraft training data and MCU evaluation protocol as GameWAM, achieving 25.2% versus our 46.6% MCU All ASR—a 21.4-percentage-point improvement. Our evaluation further spans MCU Mini and MCU All, four ViZDoom maps, and zero-shot VoxeLibre transfer (59.2% ASR), alongside component ablations and causal LASI analyses. Together, these experiments provide both controlled methodological evidence and comprehensive system-level evaluation. Given this breadth, we believe the paper already provides substantial support for its claims and would welcome specific suggestions for additional experiments that would materially change this assessment.

AuthorsOpenReview ↗

Response to Reviewer GoLV (Part 3/4)

W3 / Q2. ViZDoom expert performance and controlled baselines

We thank the reviewer for this constructive suggestion and agree that reporting the original APPO experts alongside a controlled baseline would strengthen our ViZDoom evaluation. We therefore evaluate APPO under matched random seeds and environmental conditions. Since the suggested behavior-cloning baseline does not specify an architecture, we select Fast-WAM as an established and methodologically relevant WAM baseline. Training both models on the same curated demonstrations and evaluating them under the same protocol provides a more direct comparison of their world–action formulations:

ScenarioAPPO ExpertFast-WAMGameWAM
Battle 142.32 ± 6.2539.70 ± 7.6743.12 ± 5.48
Battle 216.60 ± 6.3814.58 ± 7.2619.02 ± 5.84
Defend the Center15.22 ± 2.9512.10 ± 2.5615.36 ± 1.63
Defend the Line18.48 ± 6.7315.46 ± 5.5223.52 ± 5.76

GameWAM achieves higher mean rewards than the original APPO experts on all four maps. This is plausible because GameWAM is trained on quality-filtered APPO demonstrations, which emphasize higher-quality trajectories rather than the full distribution of the original expert's behavior. More importantly, GameWAM consistently outperforms Fast-WAM on identical curated data, indicating that its performance advantage cannot be explained by data filtering alone. We also recognize that Game-TARS is evaluated zero-shot in ViZDoom, but its reported overall training-token scale is approximately 200× ours (Appendix D.1). Generalist-agent comparisons thus measure system capability under unequal training resources, whereas the matched Fast-WAM comparison more directly tests our formulation.

W4 / Q3. Step counting and success–steps analysis

We thank the reviewer for this constructive suggestion and agree that steps averaged only over successful episodes do not fully capture the efficiency–success trade-off. We would first like to clarify that, for a fair comparison with the MCU baselines, we follow OpenHA’s steps-to-completion convention. We count each executed native action as one step and average the step counts over successful episodes within each task category (Appendix C.6). To provide a more comprehensive evaluation, we additionally report ASR under different executed-action budgets, counting all evaluated episodes in the denominator:

Executed-action budgetMCU Mini ASRMCU All ASR
10022.0%16.5%
20040.0%36.5%
30047.3%41.5%
40048.7%43.9%
50050.3%45.6%
60050.7%46.6%

Notably, MCU All reaches 43.9% ASR within 400 actions, compared with 46.6% at the 600-action limit.

W4 / Q5. Ablation reliability and statistical interpretation

We apologize that the original table captions did not clearly define uncertainty and have clarified them. As specified in Appendix C.6 and consistent with the MCU/OpenHA convention, the reported ±SD in both main benchmark and ablation tables is computed across per-task success rates, not across training or evaluation seeds. Each ablation configuration is evaluated on 30 MCU Mini tasks with 10 distinct evaluation seeds per task, totaling 300 closed-loop episodes. Each task's success rate is computed over its 10 episodes, and SD across the 10 task-level success rates within each category. Thus, large SD reflects heterogeneous task difficulty and agent competence, not training instability. For example, GameWAM succeeds in all 10 episodes when harvesting sugar cane but none when collecting birch logs, contributing to a large task-wise SD. Conversely, a model performing poorly across all tasks may show a small SD because its success rates are uniformly low. To further address the reviewer's concern about statistical significance, we compute 95% confidence intervals using category-stratified, task-paired bootstrap resampling (200,000 resamples):

AblationASR drop (pp)95% CI (pp)
Action-only supervision15.0[8.3, 21.7]
Coarser temporal sampling14.0[6.7, 21.7]
No event-anchored sampling12.7[6.0, 19.3]
Unified action distribution12.3[7.0, 17.7]
Matched prediction–execution horizon9.3[3.3, 15.7]
No cross-cycle history4.0[−1.7, 9.7]

Five of the six ablations have 95% confidence intervals excluding zero, supporting the robustness of these improvements to task resampling. The cross-cycle-history CI includes zero because its effects vary across categories: GUI and Combat improve by 12.0 and 5.0 percentage points, respectively, while Embodied decreases by 5.0 points. Consequently, the overall improvement of 4.0 points can fluctuate under task resampling, reflecting the task-dependent contribution of cross-cycle history, particularly for GUI interactions, rather than a uniform benefit across all tasks.

AuthorsOpenReview ↗

Response to Reviewer GoLV (Part 4/4)

W5 / Q4. LASI mechanism and reuse versus resampling

We respectfully disagree with the reviewer's premise on two counts. First, fixed action-noise sources are not inherently unconventional: reuse also appears in existing WAM implementations, including DreamZero and Fast-WAM. Second—and more importantly—LASI is not caused by noise reuse. Our donor-swap, low-frequency zeroing, antithetic, and permutation interventions identify an intrinsic, frequency-specific causal coupling between sampled action-source components and coarse camera motion, independently of whether the source is reused. Reuse simply preserves a source-induced directional bias over replanning cycles, turning an intrinsic model sensitivity into persistent closed-loop failures.

To quantify that consequence, we additionally compare fixed-source reuse with GameWAM's standard resampling at each replanning step:

Action sourceEmbodied MiniEmbodied AllGUI MiniGUI AllCombat MiniCombat AllAvg. MiniAvg. All
Fixed-source reuse2.00.338.056.81.00.813.719.3
Resampling70.047.543.060.039.032.250.746.6

Fixed-source reuse reduces average ASR by 37.0 points on MCU Mini and 27.3 points on MCU All. Embodied and Combat performance nearly collapses, whereas GUI performance remains comparatively preserved, consistent with LASI affecting continuous camera controls. These results tie the causal diagnostics to substantial real-task harm, rather than describing only an unusual sampling artifact. As discussed in Appendix E.2, low-frequency structure may also arise in physical embodied actions; similar source-induced bias could matter there, though we do not claim to have tested this transfer.

Q6. Future-video prediction and its contribution to closed-loop control

We thank the reviewer for raising this important question and appreciate the opportunity to clarify the role of world modeling in GameWAM. We respectfully believe this concern partly reflects a misunderstanding of the objective of an action-oriented WAM versus a standalone world model. GameWAM is fundamentally an action model whose goal is to learn an executable policy for closed-loop game control. World modeling provides predictive supervision for this policy. Recent work further shows that explicit future-video generation is not necessary for effective world–action control. Fast-WAM skips test-time video prediction, ForeWAM uses latent future representations without video decoding, and DELE-w0.5 learns compact future-state latents instead of videos. Pixel-level video quality is thus neither a universally applicable nor a sufficient measure of action-relevant world modeling. This is why WAM studies such as DreamZero and Fast-WAM place primary weight on closed-loop action performance. Following this practice, we evaluate how effectively GameWAM’s actions accomplish the instructed tasks in closed-loop interaction. We report task success and steps to completion on MCU and mean episode reward on ViZDoom. Like Fast-WAM, GameWAM generates no future frames during online rollout, so there are no predicted frames to compare with subsequent observations.

Nevertheless, to directly address the reviewer’s question about prediction quality, we evaluated future-video prediction offline on 350 samples, obtaining 29.50 dB PSNR and 0.7726 SSIM against the recorded future frames. These metrics quantify pixel-level and structural similarity, respectively. A small viewpoint shift can lower both scores even when the task-relevant content is preserved. Conversely, a predicted inventory screen can remain visually close to the reference even if an ingredient is placed in the wrong slot, an error that can determine whether crafting succeeds. To examine how world modeling contributes to control, we therefore also consider closed-loop evidence beyond the action-only ablation. In Appendix D.3, under otherwise matched settings, the joint video–action variant fits the action objective faster during training, yet its predicted future frames appear less sharp on validation examples and its closed-loop performance is lower. The default modality-decoupled model achieves 50.7% versus 46.3% ASR on MCU Mini and 46.6% versus 39.6% on MCU All (Table 7). This comparison shows that how video and action learning are coupled matters for control even when both models receive future-video supervision. The action-only ablation directly measures the benefit of that supervision. Adding it raises MCU Mini ASR from 35.7% to 50.7% (+15.0 percentage points, task-paired 95% CI [8.3, 21.7]), with gains of +10.0, +9.0, and +26.0 points on Embodied, GUI, and Combat tasks, respectively.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Global response and summary of changes in the revision

Summary of Revisions and Additional Experiments

We sincerely thank all reviewers for their thoughtful reviews and constructive feedback. We appreciate their recognition of the significance of native game control, the benefits of our modeling choices, the promising closed-loop results, and the scientific value of our LASI investigation. Their suggestions have helped us further strengthen the paper.

We have revised the manuscript and incorporated additional controlled experiments. In the highlighted manuscript, we mark the revisions summarized below in blue.

  • Methodological contributions (Section 1): We clarified the novelty of GameWAM, emphasizing WAMs as a competitive paradigm for native game control, our unified native-action and block–cycle formulation, and LASI as an independent scientific discovery in generative control. (Reviewers GoLV, 9hEk)

  • Budget-matched hierarchical history ablation (Table 2): We added a recent-only history baseline under the same 40-token context budget to distinguish the benefits of hierarchical organization from simply retaining more historical information. (Reviewer ob9D)

  • Practical impact of LASI (Section 5.5; Appendix D.7.8, Table 9): We added controlled comparisons of fixed-source reuse and per-replanning-step resampling on MCU Mini and MCU All, quantifying the substantial closed-loop performance degradation caused by source reuse. (Reviewers GoLV, ob9D)

  • ViZDoom expert and controlled baseline comparisons (Appendix D.2, Table 6): We added the original APPO experts' performance and a data-matched Fast-WAM comparison across all four ViZDoom scenarios. (Reviewer GoLV)

  • Success under executed-action budgets (Appendix D.4): We added success-rate curves across different action budgets on MCU Mini and MCU All to provide a more comprehensive view of the efficiency–success trade-off. (Reviewer GoLV)

  • Evaluation clarity and related work (Table 1, Table 2; Section 2.1): We clarified the interpretation of task-wise standard deviations in the benchmark and ablation tables and further distinguished GameWAM from WHAM in the related work. (Reviewers GoLV, 9hEk)

Detailed responses to each concern are provided under the corresponding reviews.

We sincerely thank all reviewers again for their valuable suggestions, which have helped improve the clarity and experimental support of our work. We would be happy to discuss any further questions.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.