- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper introduces a multi-agent world model for gaming and robotics data. Authors appear to only consider 2D games where the playable characters share a screen (i.e. not multi-view). They propose to adapt a WAN model to streaming inference (with block causal attention) to the task. Their core contributions are two modulation mechanisms based on semantic descriptions: visual tokens are modulated to identify the character of interest, and numerical action representations are modulated with environment specific textual descriptions. They build a dataset of 4 games (recorded on an emulator with scripted policies) and 2 robotics datasets (pre-existing datasets). Authors run comparisons to state of the art world models as well as several ablations, showing promising results.
Critical strengths
- The objective of designing a model that can handle heterogeneous types of datasets and actions is useful and a nice deviation from common large scale world models that often focus purely on navigation. It is particularly nice that the approach combines gaming and robotics dataset, while at the same time separating the two tasks in some evaluations.
- The core idea is sensible: injecting semantic information to guide the model, both in terms of character identification and action understanding.
- The paper is, most the most part, easy to read and to follow.
Critical weaknesses
- My main concern relates to the baselines considered in this work. Competing approaches seem to be evaluated in a very unfair setting: LingBot, multiworld, Incantation are directly evaluated with no adaptation. While LingBot is trained on large scale diverse data, Multiworld and Incantation are both trained/finetuned on a single 3D game (It takes two and Elden Ring), making it unlikely that they would perform well on the proposed dataset (especially Incantation), especially compared to a model trained on this exact dataset. As authors do not specify which publicly available checkpoints they are using, I am using Table 1 as reference. In addition, Multiworld is trained for multi-view settings (a task that is a little different) and seems to have available training code. ActionParty and WanToFight are fairer comparisons, since they are finetuned Wan models on the target dataset. However, authors should confirm that the same Wan model is finetuned (both use Wan 1.3B, while this work uses a 5B model). Authors should make these facts very clear in the main paper and acknowledge that most of their comparisons are unfair. If WanToFight and ActionParty do not use the same Wan backbone, this should be fixed. It would have been better to implement some of these comparisons as simple baselines: e.g. representing actions as a semantic input as LingBot does.
- Authors should differentiate their multi-agent setting to the multi-view setup used in e.g. Multiworld. The former has all agents visible on screen nearly at all time, sharing a view and interacting in the same 2D screen. The latter has separate views per player, typically involves 3D environments and partial observability, including of the other agents. Both those tasks come with different challenges (e.g. consistency between views, agent identification), and should be treated as different problems.
- The method description lacks clarity. In particular, authors do not explain the core architecture and integration of their module until the end of the method section, making it difficult to understand how the pieces work together early on. The intuition and motivation behind the methodological choices is not provided (note this is different than 3.1, which motivates the overall concept of the paper). Why modulate visual tokens and not simply provide the computed attention masks or directly use cross attention layers?
- The main reason why an elaborate action representation seems to be the mismatch between gaming and robotics data. Wouldn’t it be possible to simply unify all game controls as controller presses? Authors point out that all actions are binary button presses and therefore could be represented in a much more simpler way in the context of gaming. The approach also requires introducing more annotations (action description, agent description), which, at scale, can make training more challenging and require potentially unreliable LLM-based annotations.
- The model being real time is a nice touch, but I would not claim block causal fine-tuning with DMD of WAN as a contribution, as this is a standard strategy.
Questions for authors
- Can you confirm that the WanToFight and ActionParty backbones match the one used in this work? Can a baseline using the LingBot action representation be considered? Please clarify which checkpoints are used for the different models compared to, as checkpoints trained on different environments are available. This is my main concern with this work. If authors can demonstrate fair comparisons, I would be happy to reconsider my rating.
- The SOTA comparison table reports combined scores (games + robotics). Some competing methods have their action space designed with game controllers in mind, other have seen the robotics data (LingBot). Can authors share results on game and robotics data separately, to get a clearer picture?
- Have you tried simpler approaches such as a simple cross attention layer between semantic and visual/numerical features?
6 comments
Response to Reviewer hMBV (2/2)
W4: Unifying game controls as controller buttons; annotation cost and LLM noise
A4: Indeed, across all four games, our raw 12D action vectors directly use the unified 12 Genesis controller buttons (Appendix. B.1). However, unifying at the button-index level alone is insufficient because the same physical button triggers conflicting visual dynamics across games (e.g., index 10 is punch in fighters vs. accelerate in Micro Machines). Keeping button indices without semantics (ID tokens, Appendix. Table 5) drops Game Act-F1 to 0.230 and Attr-Acc to 0.588, whereas permuting dimensions and schema texts together into an unseen layout preserves 0.592 / 0.897. Regarding annotation cost and LLM reliability (Fig. 4): action schemas are defined only once per environment (six static 12-line tables), and agent identities come deterministically from simulator states (with VLM/LLM only used to phrase scene/character templates rather than noisy per-frame action labeling). Moreover, paraphrasing agent descriptions at inference preserves 0.596 Act-F1 and 0.895 Attr-Acc (see Table 2 for Reviewer yXwe), confirming high robustness to text variations.
W5: Real-time generation as a contribution
A5: Thank you for the valuable suggestion. Agreed. We will revise Sec. 1 to remove block-causal + DMD as a standalone contribution and focus on preserving HAI's multi-agent control capability through causal distillation (Appendix B.2, Appendix Table 3).
Q2: Reporting game and robotics results separately
A6: Table 2 already reports Games (top) and Robotics (bottom) separately, with no combined score. We will bold both headers in Table 2 in the final version.
Official Comment to Reviewer hMBV
Dear Reviewer hMBV,
We thank you for your time and effort in reviewing our paper. We have responded to each of your concerns in our rebuttal and updated PDF at the following locations:
W1 & Q1(Fair 5B Comparisons & Checkpoints) — resolved inA1(Response Table 1) andAppendix F.7 (Fig. 8): Clarified all baseline checkpoints and retrained MultiWorld, ActionParty, and Text-Action (LingBot-style) on our dataset using the same Wan2.2-TI2V-5B backbone, where LiveAct consistently outperforms all retrained baselines.W2, W5 & Q2(Problem Scope, Contributions & Separate Scores) — resolved inA2,A5, andA6(with details inSec. 1,Sec. 2,Table 2, andAppendix E.2): Clarified shared-view multi-agent vs. multi-view settings, refocused causal distillation on preserving HAI control, and noted that Table 2 reports Games and Robotics separately.W3, W4 & Q3(Method Clarity, Controller Schema & Simpler Cross-Attention) — resolved inA3(Response Table 2),A4, andAppendix F.3 (Fig. 2): Compared against simpler cross-attention variants and showed that button-index unification fails across games (0.588Attr-Acc) whereas LiveAct generalizes to unseen button layouts (0.897Attr-Acc) and paraphrased descriptions (0.895Attr-Acc) with minimal annotation cost.
If you have any unresolved issues or new questions, please let us know. We are more than happy to answer any further questions you may have!
Thank you once again for your consideration.
Best regards,
Authors
Response to Reviewer hMBV (1/2)
Thank you for your comprehensive review of our paper and for noting that you would reconsider your rating with fair comparisons. We provide our feedback as follows, along with additional qualitative comparisons in the revised appendix.
A1: Thank you for pointing this out. In Table 2 of the submitted paper, WanToFight-style already used Wan2.2-TI2V-5B, ActionParty used its native Wan2.1-1.3B, and zero-shot LingBot-World 2.0 (14B), MultiWorld (Wan2.2-5B), and Incantation (Wan2.2-5B) used their official released weights. For a strictly fair comparison, we retrained MultiWorld (with its official code), ActionParty, and a LingBot-style text-action baseline on our dataset using the same Wan2.2-TI2V-5B backbone as LiveAct. As shown in Table 1 below and Appendix Fig. 8 (Appendix F.7), LiveAct consistently outperforms all retrained 5B baselines across both games and robotics without requiring initial coordinates or identity crops. We will include these quantitative results and all baseline checkpoint details in the final version.
Table 1. Fair comparison on the same dataset and video backbone (Wan2.2-TI2V-5B).
A2: Thank you for the suggestion. Agreed that this distinction should be highlighted more explicitly. In the submitted paper, we already frame our task around a shared visual future / scene in Sec. 1 (Lines 63–66, 89–90), separate multi-observation models (Solaris, MultiWorld) from entity-binding methods in Sec. 2 (Lines 152–161), and note in Table 2 (Lines 433–434) and Appendix E.2 (Lines 934–935) that native MultiWorld relies on synchronized multi-view observations while our setting uses a single shared perspective. To make this contrast unmistakable up front, we will revise Sec. 1 and Sec. 2 to explicitly distinguish our shared-view multi-agent setting (binding concurrent actions to co-occurring entities on a shared screen) from multi-view settings (cross-view consistency under partial observability).
A3: Valuable question! We will add a data-flow overview and module motivations at the start of Sec. 3.
w/o AASM,w/o SGAA, orAdditivein Table 3 and Appendix Table 5) severely degrades control and attribution (Table 2).Table 2. Comparison with simpler cross-attention variants.
w/o AASM(Plain Visual–Action CA, Table 3)w/o SGAA(Flat Action CA, Table 3)