acceptodds
Under review as a conference paper at ICLR 2027

LiveAct: Towards Real-Time Multi-Agent World Models with Heterogeneous Action Interfaces

Abstract

Interactive video world models predict how an environment evolves under user actions, but most existing systems assume one controlled agent and one task-specific action interface. We study multi-agent video world modeling, where one model generates a joint visual future for multiple controlled agents while interpreting heterogeneous controls ranging from binary game buttons to continuous robot commands. This task presents two complementary challenges. First, every agent-indexed action stream must control its intended visual entity independently. Second, the model must adapt the meaning of each control variable across different tasks, where the same tensor dimension may denote an attack, steering command, or robot command. We introduce LiveAct, centered on a Heterogeneous Action Interface (HAI) whose two components mirror this decomposition. To support independent control, Agent-Aware Spatial Modulation uses language-driven spatial gates and agent-specific attention to bind each action stream to its intended visual agent. To accommodate heterogeneous controls, Schema-Guided Action Adaptation describes each control variable in language and modulates its semantic token with the synchronized numerical value. We further transfer the complete interface to a blockwise causal generator through teacher forcing and rollout-aware distribution matching, enabling efficient streaming generation while retaining agent-specific control and action semantics. We instantiate the formulation on six multi-agent sources spanning games and multi-robot manipulation, and evaluate visual quality, action following, agent attribution, and latency. Across games and robot manipulation, LiveAct reaches 0.898 and 0.942 agent-attribution accuracy without agent locations or identity crops, and the causal generator attains 17.70 FPS at .

Primary area:
generative models
Keywords:
Multi-Agent World Model, Diffusion Model, Interactive Video Generation

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_hMBVOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces a multi-agent world model for gaming and robotics data. Authors appear to only consider 2D games where the playable characters share a screen (i.e. not multi-view). They propose to adapt a WAN model to streaming inference (with block causal attention) to the task. Their core contributions are two modulation mechanisms based on semantic descriptions: visual tokens are modulated to identify the character of interest, and numerical action representations are modulated with environment specific textual descriptions. They build a dataset of 4 games (recorded on an emulator with scripted policies) and 2 robotics datasets (pre-existing datasets). Authors run comparisons to state of the art world models as well as several ablations, showing promising results.

Critical strengths
  • The objective of designing a model that can handle heterogeneous types of datasets and actions is useful and a nice deviation from common large scale world models that often focus purely on navigation. It is particularly nice that the approach combines gaming and robotics dataset, while at the same time separating the two tasks in some evaluations.
  • The core idea is sensible: injecting semantic information to guide the model, both in terms of character identification and action understanding.
  • The paper is, most the most part, easy to read and to follow.
Critical weaknesses
  • My main concern relates to the baselines considered in this work. Competing approaches seem to be evaluated in a very unfair setting: LingBot, multiworld, Incantation are directly evaluated with no adaptation. While LingBot is trained on large scale diverse data, Multiworld and Incantation are both trained/finetuned on a single 3D game (It takes two and Elden Ring), making it unlikely that they would perform well on the proposed dataset (especially Incantation), especially compared to a model trained on this exact dataset. As authors do not specify which publicly available checkpoints they are using, I am using Table 1 as reference. In addition, Multiworld is trained for multi-view settings (a task that is a little different) and seems to have available training code. ActionParty and WanToFight are fairer comparisons, since they are finetuned Wan models on the target dataset. However, authors should confirm that the same Wan model is finetuned (both use Wan 1.3B, while this work uses a 5B model). Authors should make these facts very clear in the main paper and acknowledge that most of their comparisons are unfair. If WanToFight and ActionParty do not use the same Wan backbone, this should be fixed. It would have been better to implement some of these comparisons as simple baselines: e.g. representing actions as a semantic input as LingBot does.
  • Authors should differentiate their multi-agent setting to the multi-view setup used in e.g. Multiworld. The former has all agents visible on screen nearly at all time, sharing a view and interacting in the same 2D screen. The latter has separate views per player, typically involves 3D environments and partial observability, including of the other agents. Both those tasks come with different challenges (e.g. consistency between views, agent identification), and should be treated as different problems.
  • The method description lacks clarity. In particular, authors do not explain the core architecture and integration of their module until the end of the method section, making it difficult to understand how the pieces work together early on. The intuition and motivation behind the methodological choices is not provided (note this is different than 3.1, which motivates the overall concept of the paper). Why modulate visual tokens and not simply provide the computed attention masks or directly use cross attention layers?
  • The main reason why an elaborate action representation seems to be the mismatch between gaming and robotics data. Wouldn’t it be possible to simply unify all game controls as controller presses? Authors point out that all actions are binary button presses and therefore could be represented in a much more simpler way in the context of gaming. The approach also requires introducing more annotations (action description, agent description), which, at scale, can make training more challenging and require potentially unreliable LLM-based annotations.
  • The model being real time is a nice touch, but I would not claim block causal fine-tuning with DMD of WAN as a contribution, as this is a standard strategy.
Questions for authors
  • Can you confirm that the WanToFight and ActionParty backbones match the one used in this work? Can a baseline using the LingBot action representation be considered? Please clarify which checkpoints are used for the different models compared to, as checkpoints trained on different environments are available. This is my main concern with this work. If authors can demonstrate fair comparisons, I would be happy to reconsider my rating.
  • The SOTA comparison table reports combined scores (games + robotics). Some competing methods have their action space designed with game controllers in mind, other have seen the robotics data (LingBot). Can authors share results on game and robotics data separately, to get a clearer picture?
  • Have you tried simpler approaches such as a simple cross attention layer between semantic and visual/numerical features?
6 comments
AuthorsOpenReview ↗

Response to Reviewer hMBV (1/2)

Thank you for your comprehensive review of our paper and for noting that you would reconsider your rating with fair comparisons. We provide our feedback as follows, along with additional qualitative comparisons in the revised appendix.


W1 & Q1: Fair baseline comparisons, backbone consistency, and checkpoints

A1: Thank you for pointing this out. In Table 2 of the submitted paper, WanToFight-style already used Wan2.2-TI2V-5B, ActionParty used its native Wan2.1-1.3B, and zero-shot LingBot-World 2.0 (14B), MultiWorld (Wan2.2-5B), and Incantation (Wan2.2-5B) used their official released weights. For a strictly fair comparison, we retrained MultiWorld (with its official code), ActionParty, and a LingBot-style text-action baseline on our dataset using the same Wan2.2-TI2V-5B backbone as LiveAct. As shown in Table 1 below and Appendix Fig. 8 (Appendix F.7), LiveAct consistently outperforms all retrained 5B baselines across both games and robotics without requiring initial coordinates or identity crops. We will include these quantitative results and all baseline checkpoint details in the final version.

Table 1. Fair comparison on the same dataset and video backbone (Wan2.2-TI2V-5B).

MethodsExtra InputGame FVD↓Game Act-F1↑Game Attr-Acc↑Robot FVD↓Robot NRMSE↓Robot Attr-Acc↑
MultiWorld (retrained)None516.40.2360.566355.60.6580.894
ActionParty (retrained)Init. coords902.40.1480.576911.01.0500.642
Text-Action (LingBot-style)None512.50.3190.658327.70.6460.874
WanToFight-styleID crops563.60.2410.552402.90.7580.886
LiveAct (Ours)None338.50.5940.898270.20.5580.942
  • Compared with MultiWorld and ActionParty: MultiWorld's index conditioning works for fixed-base robots (0.894 Attr-Acc) but fails on freely moving game characters (0.566) due to the lack of semantic–visual binding. LiveAct outperforms both retrained 5B baselines without needing initial coordinates or ID crops.
  • Compared with Text-Action: Serializing multi-agent trajectories as text hits two bottlenecks: (1) prompt length quickly exceeds umT5's 512-token limit (>2,000 tokens for 81 frames), forcing aggressive action subsampling and truncation; (2) streaming requires re-encoding text prompts at every chunk. LiveAct avoids both by caching static semantic embeddings once and feeding online action values through a lightweight MLP.

W2: Distinguishing multi-agent from multi-view settings

A2: Thank you for the suggestion. Agreed that this distinction should be highlighted more explicitly. In the submitted paper, we already frame our task around a shared visual future / scene in Sec. 1 (Lines 63–66, 89–90), separate multi-observation models (Solaris, MultiWorld) from entity-binding methods in Sec. 2 (Lines 152–161), and note in Table 2 (Lines 433–434) and Appendix E.2 (Lines 934–935) that native MultiWorld relies on synchronized multi-view observations while our setting uses a single shared perspective. To make this contrast unmistakable up front, we will revise Sec. 1 and Sec. 2 to explicitly distinguish our shared-view multi-agent setting (binding concurrent actions to co-occurring entities on a shared screen) from multi-view settings (cross-view consistency under partial observability).


W3 & Q3: Method clarity; why not attention masks or plain cross-attention?

A3: Valuable question! We will add a data-flow overview and module motivations at the start of Sec. 3.

  • Attention masks: Require inference-time boxes or coordinates, which we eliminate; AASM learns soft spatial gating end-to-end without spatial supervision (as shown in Appendix Fig. 2, Appendix F.3).
  • Plain cross-attention: HAI is per-agent cross-attention between visual queries and action keys/values (Eq. 6), where AASM modulates queries to locate each agent and SGAA modulates keys/values with channel semantics. Replacing either modulation with plain cross-attention (w/o AASM, w/o SGAA, or Additive in Table 3 and Appendix Table 5) severely degrades control and attribution (Table 2).

Table 2. Comparison with simpler cross-attention variants.

MethodsGame Act-F1↑Game Attr-Acc↑Robot NRMSE↓Robot Attr-Acc↑
w/o AASM (Plain Visual–Action CA, Table 3)0.1720.5920.9310.810
w/o SGAA (Flat Action CA, Table 3)0.2250.6031.1200.484
Additive Semantic + Numerical CA (Appendix Table 5)0.2090.5131.0710.480
LiveAct (Full HAI)0.5940.8980.5580.942
AuthorsOpenReview ↗

Response to Reviewer hMBV (2/2)

W4: Unifying game controls as controller buttons; annotation cost and LLM noise

A4: Indeed, across all four games, our raw 12D action vectors directly use the unified 12 Genesis controller buttons (Appendix. B.1). However, unifying at the button-index level alone is insufficient because the same physical button triggers conflicting visual dynamics across games (e.g., index 10 is punch in fighters vs. accelerate in Micro Machines). Keeping button indices without semantics (ID tokens, Appendix. Table 5) drops Game Act-F1 to 0.230 and Attr-Acc to 0.588, whereas permuting dimensions and schema texts together into an unseen layout preserves 0.592 / 0.897. Regarding annotation cost and LLM reliability (Fig. 4): action schemas are defined only once per environment (six static 12-line tables), and agent identities come deterministically from simulator states (with VLM/LLM only used to phrase scene/character templates rather than noisy per-frame action labeling). Moreover, paraphrasing agent descriptions at inference preserves 0.596 Act-F1 and 0.895 Attr-Acc (see Table 2 for Reviewer yXwe), confirming high robustness to text variations.


W5: Real-time generation as a contribution

A5: Thank you for the valuable suggestion. Agreed. We will revise Sec. 1 to remove block-causal + DMD as a standalone contribution and focus on preserving HAI's multi-agent control capability through causal distillation (Appendix B.2, Appendix Table 3).


Q2: Reporting game and robotics results separately

A6: Table 2 already reports Games (top) and Robotics (bottom) separately, with no combined score. We will bold both headers in Table 2 in the final version.

AuthorsOpenReview ↗

Official Comment to Reviewer hMBV

Dear Reviewer hMBV,

We thank you for your time and effort in reviewing our paper. We have responded to each of your concerns in our rebuttal and updated PDF at the following locations:

  • W1 & Q1 (Fair 5B Comparisons & Checkpoints) — resolved in A1 (Response Table 1) and Appendix F.7 (Fig. 8): Clarified all baseline checkpoints and retrained MultiWorld, ActionParty, and Text-Action (LingBot-style) on our dataset using the same Wan2.2-TI2V-5B backbone, where LiveAct consistently outperforms all retrained baselines.
  • W2, W5 & Q2 (Problem Scope, Contributions & Separate Scores) — resolved in A2, A5, and A6 (with details in Sec. 1, Sec. 2, Table 2, and Appendix E.2): Clarified shared-view multi-agent vs. multi-view settings, refocused causal distillation on preserving HAI control, and noted that Table 2 reports Games and Robotics separately.
  • W3, W4 & Q3 (Method Clarity, Controller Schema & Simpler Cross-Attention) — resolved in A3 (Response Table 2), A4, and Appendix F.3 (Fig. 2): Compared against simpler cross-attention variants and showed that button-index unification fails across games (0.588 Attr-Acc) whereas LiveAct generalizes to unseen button layouts (0.897 Attr-Acc) and paraphrased descriptions (0.895 Attr-Acc) with minimal annotation cost.

If you have any unresolved issues or new questions, please let us know. We are more than happy to answer any further questions you may have!

Thank you once again for your consideration.

Best regards,

Authors

Reviewer_aJMVOpenReview ↗
Rating
3
Confidence
2
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

This paper proposes LiveAct, a real-time multi-agent video world model supporting heterogeneous action interfaces across multiplayer games and multi-robot manipulation. The proposed Heterogeneous Action Interface (HAI) combines Agent-Aware Spatial Modulation (AASM) for binding action streams to their corresponding visual agents and Schema-Guided Action Adaptation (SGAA) for integrating action semantics with numerical control values. A three-stage training strategy enables efficient causal video generation. Experiments across six environments demonstrate improvements in visual quality, action following, agent attribution, and generation efficiency, achieving 17.70 FPS at 512 × 448 resolution.

Critical strengths
  1. The proposed Heterogeneous Action Interface provides a well-motivated and effective approach to multi-agent control across heterogeneous action spaces. The combination of AASM and SGAA is technically meaningful and supported by comprehensive ablation studies.

  2. The experiments demonstrate strong performance across multiplayer games and multi-robot manipulation, with improvements in visual quality, action following, and agent attribution. The real-time generation capability further enhances the practical significance of the approach.

Critical weaknesses
  1. Although LiveAct demonstrates qualitative results with up to four agents, the paper lacks quantitative evaluations of how action-following accuracy, agent attribution, and inference efficiency vary with the number of agents. This limits the assessment of its multi-agent scalability.

  2. The robustness of agent-specific control under challenging interactions, such as visually similar agents, occlusion, and spatial overlap, is not systematically evaluated. Such analyses would provide stronger evidence for the reliability of AASM.

Questions for authors
  1. Could the authors provide quantitative results for different numbers of agents (e.g., 2, 3, and 4), including action-following accuracy, agent attribution, and inference speed?

  2. How robust is AASM when agents have similar appearances, overlap spatially, or occlude one another? Are there additional quantitative results for these challenging scenarios?

3 comments
AuthorsOpenReview ↗

Response to Reviewer aJMV

Thank you for your comprehensive and detailed review of our paper and the recognition of our work's clarity and effectiveness. We provide our feedback as follows.


W1 & Q1: Quantitative evaluation across 2, 3, and 4 agents

A1: Thank you for the constructive suggestion. Each additional agent adds one agent description, one action stream, and one per-agent cross-attention residual in Eq. 7 without modifying the HAI architecture (Appendix F.5, Appendix Fig. 6). We evaluate action-following accuracy, agent attribution, and single-A800 inference speed across 2, 3, and 4 agents in Table 1 (qualitative results are shown in Appendix Fig. 9, Appendix F.8).

Table 1. Scalability across different numbers of agents.

#Agents (NN)Game Act-F1↑Game Attr-Acc↑Inference Speed (FPS)↑
20.5940.89817.7
30.5640.74714.2
40.5980.72511.9

As NN increases from 2 to 3 and 4, inference speed decreases mildly (~20% and ~33%) while maintaining interactive frame rates, and Act-F1 remains stable across all agent counts. Attr-Acc also matches the ground-truth evaluator reference on each subset (Bomberman achieves 0.995–0.998 Attr-Acc across 2P/3P/4P, while 3P/4P averages include top-down Micro Machines with tiny vehicle sprites; 3-arm robot manipulation likewise achieves >0.94 Attr-Acc).


W2 & Q2: Robustness under similar appearance, spatial overlap, and occlusion

A2: Good question! Beyond our bimanual robot tasks (two visually identical arms with 0.942 Attr-Acc) and the per-agent attention heatmaps in Appendix Fig. 2 (Appendix F.3), we perform a frame-level breakdown across all 78 game test clips (6,240 frames). We group frames by inter-agent bounding-box overlap ratio (intersection over smaller box area) to evaluate spatial overlap and occlusion, and separately evaluate Mega Bomberman, where agents share identical sprite shapes and differ only in color (Table 2 and Appendix Fig. 9, Appendix F.8).

Table 2. Frame-level evaluation under spatial overlap, occlusion, and similar appearance.

Interaction Subset#FramesGame Act-F1↑Game Attr-Acc↑
No overlap5,6320.5960.887
Partial overlap (< 20% box area)3040.7230.939
Heavy overlap / occlusion (≥ 20% box area)3040.6201.000
Visually similar agents (Mega Bomberman)1,2000.6600.994

Both action following and attribution remain strong under partial overlap, heavy occlusion, and visually similar appearances without degrading relative to non-overlapping frames (overlapping frames mainly occur during close-range fighting-game combat). We will include these results in the final version.

AuthorsOpenReview ↗

Official Comment to Reviewer aJMV

Dear Reviewer aJMV,

We thank you for your time and effort in reviewing our paper, and we are pleased that you acknowledged its clear motivation and strong performance. We have responded to each of your concerns in our rebuttal and updated PDF at the following locations:

  • W1 & Q1 (Scalability Across 2, 3, and 4 Agents) — resolved in A1 (Response Table 1) and Appendix F.8 (Fig. 9a–b): Added quantitative evaluations of control accuracy (Act-F1), attribution (Attr-Acc), and inference speed (FPS) for 2, 3, and 4 agents, showing stable action following and interactive frame rates (17.7 → 14.2 → 11.9 FPS).
  • W2 & Q2 (Robustness to Overlap, Occlusion, and Similar Appearance) — resolved in A2 (Response Table 2) and Appendix F.8 (Fig. 9a, c): Added a frame-level breakdown across all 6,240 test frames, confirming strong control and attribution under partial overlap (0.939 Attr-Acc), heavy occlusion (1.000), and visually similar agents (0.994 on Mega Bomberman, 0.942 on identical robot arms).

If you have any unresolved issues or new questions, please let us know. We are more than happy to answer any further questions you may have!

Thank you once again for your consideration.

Best regards,

Authors

AuthorsOpenReview ↗

Following Up Response to Reviewer aJMV

Dear Reviewer,

We sincerely apologize for following up again. With less than 24 hours remaining before the discussion deadline, we would greatly appreciate it if you could take a moment to review our rebuttal and share any further feedback.

We have carefully responded to your concerns and would be happy to provide additional clarifications if needed. Your feedback would be invaluable to us.

Thank you very much for your time and consideration.

Best regards,

The Authors

Reviewer_yXweOpenReview ↗
Rating
3(+1)
Confidence
1
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

A world model that can generate subsequent frames in real time, jointly conditioned on the actions of multiple characters. The HAI module includes role-aware spatial modulation and schema-guided action adaptation. Results are much better on the test set.

Critical strengths

Designed an interface for multi-character video world models that explicitly separates the action recipient from the action’s semantics and numerical values.

Critical weaknesses
  1. The videos are pretty short, only 81 frames.
  2. No experiment results on generalization.
  3. Lack of experiments about simulator(How can the model improve the training of the real robotic tasks)
Questions for authors
  1. How well does LiveAct generalize to unseen environments, agents, and action schemas?
  2. Can LiveAct serve as an effective simulator for policy training or planning, and do the resulting policies improve performance on real-world robotic tasks?
6 comments
AuthorsOpenReview ↗

Response to Reviewer yXwe (1/2)

Thank you for your comprehensive and detailed review of our paper and the recognition of our work's clarity and effectiveness. We provide our feedback as follows.


W1: Video length (81 frames) and long-horizon generation

A1: Thank you. 81 frames is only the default evaluation window in the main paper. Anchoring a rolling KV cache on the clean initial latent with global action–latent alignment (Sec. 3.5, Appendix B.2–B.3), LiveAct supports continuous long-horizon streaming (Appendix F.6, Appendix Fig. 7). We evaluate 241-frame rollouts across three consecutive 80-frame segments on all 78 game test clips against the retrained 5B baselines in Table 1 and Appendix Fig. 10 (Appendix F.9).

Table 1. Per-segment evaluation on 241-frame rollouts.

Frame SegmentMethodsFVD↓PSNR↑Game Act-F1↑Game Attr-Acc↑
1–80MultiWorld (retrained 5B)516.418.070.2360.566
1–80Text-Action (LingBot-style 5B)512.517.810.3190.658
1–80LiveAct (Ours)338.520.170.5940.898
81–160MultiWorld (retrained 5B)842.215.410.0830.486
81–160Text-Action (LingBot-style 5B)713.615.220.1030.510
81–160LiveAct (Ours)440.416.700.5650.790
161–240MultiWorld (retrained 5B)880.514.660.0680.528
161–240Text-Action (LingBot-style 5B)720.714.870.1810.470
161–240LiveAct (Ours)563.115.240.4690.802

LiveAct maintains visual quality, control, and attribution across all 241 frames, closely tracking ground-truth references. Conversely, MultiWorld drifts severely after frame 80, and Text-Action overflows umT5's 512-token window, dropping attribution to chance (~0.50).


W2 & Q1: Generalization to unseen control layouts, agent phrasings, and visual edits

A2: Valuable question! We evaluate zero-shot generalization across three settings:

  • Unseen control layouts: permuting the 12 action dimensions and schema descriptions together;
  • Unseen agent phrasings: paraphrasing agent descriptions into unseen wordings;
  • Unseen visual edits: editing the conditioning first frame with Qwen-Image-Edit to introduce unseen stage backgrounds and character outfits.

We also report an evaluator-free motion timing correlation rr on fighting games in Table 2 (qualitative comparisons are shown in Appendix Fig. 11, Appendix F.10).

Table 2. Zero-shot generalization across control layouts, text phrasings, and visual edits.

Zero-Shot SettingGame Act-F1↑Game Attr-Acc↑Fighting Act-F1↑Fighting Attr-Acc↑Motion Corr. rr↑Robot NRMSE↓Robot Attr-Acc↑
Original Inputs (Table 2)0.5940.8980.7150.9810.70 (GT)0.5580.942
Unseen Control Layouts0.5920.8970.7140.980—0.5620.935
Unseen Agent Phrasings0.5960.8950.7200.974—0.5560.950
Unseen Stage Backgrounds0.4550.8120.6610.9390.71——
Unseen Character Appearances0.4640.7870.6680.9330.85——

LiveAct matches original performance on unseen layouts and phrasings. On visual edits, although pixel-level IDM is sensitive to color shifts on tiny top-down sprites, LiveAct preserves the edited appearances with strong fighting-game control (0.661–0.668 Act-F1, >0.93 Attr-Acc) and high motion synchronization (r=0.71–0.85r=0.71\text{--}0.85) as shown in Appendix Fig. 11. We will include these quantitative results in the final version.

AuthorsOpenReview ↗

Response to Reviewer yXwe (2/2)

W3 & Q2: Use as a visual simulator for robot planning

A3: Interesting question! To evaluate LiveAct as a visual simulator for test-time robot planning (following WorldLine [1]), we adopt a generate-then-select pipeline: given the initial frame and N=4N=4 candidate bimanual action trajectories (e.g., sampled from π0.5\pi_{0.5} [3]), LiveAct (4-step causal generator) simulates a visual rollout for each candidate, and Qwen3-VL-8B [2] scores the rollouts (0–100, with pairwise tie-breaking) to select the best trajectory before execution (Table 3 and Appendix Fig. 12, Appendix F.11).

Table 3. Test-time robot trajectory selection using LiveAct as a visual simulator.

MethodsHandover↑Bread↑Burger↑Can↑Bowls↑RoboTwin Avg.↑RoboFactory StackCube↑
Direct Execution8.3%51.7%70.0%11.7%66.7%41.7%25.0%
LiveAct + VLM20.0%53.3%73.3%26.7%80.0%50.7% (+9.0%)81.2% (+56.2%)

LiveAct previews improve average task success by +9.0% (41.7% → 50.7%) across the five RoboTwin tasks, and boost success from 25.0% to 81.2% (+56.2%) on RoboFactory StackCube (with 90.6% pairwise discrimination between successful and failed actions). Because the RoboFactory dataset covers both successful and failed demonstrations, LiveAct accurately simulates failure modes (e.g., missed grasps and dropped cubes, as shown in Appendix Fig. 12) for poor candidate actions rather than hallucinating successful outcomes. Closed-loop policy training and real-robot deployment are our next steps.

[1] Zheng, Shenghe, et al. "WorldLine: Action-Driven Visual Simulation for Robotic Manipulation." arXiv preprint arXiv:2609.38059 (2026).

[2] Bai, Shuai, et al. "Qwen3-VL Technical Report." arXiv preprint arXiv:2511.21631 (2025).

[3] Black, Kevin, et al. "π0.5: a Vision-Language-Action Model with Open-World Generalization." CoRL (2025).

AuthorsOpenReview ↗

Official Comment to Reviewer yXwe

Dear Reviewer yXwe,

We thank you for your time and effort in reviewing our paper, and we are pleased that you recognized our multi-agent action interface design. We have responded to each of your concerns in our rebuttal and updated PDF at the following locations:

  • W1 (241-Frame Long-Horizon Generation) — resolved in A1 (Response Table 1) and Appendix F.9 (Fig. 10): Evaluated 241-frame streaming rollouts across consecutive 80-frame segments against retrained 5B baselines, showing that LiveAct sustains visual quality, control, and attribution over long horizons.
  • W2 & Q1 (Zero-Shot Generalization) — resolved in A2 (Response Table 2) and Appendix F.10 (Fig. 11): Evaluated unseen control layouts, paraphrased agent descriptions, and first-frame visual edits with unseen backgrounds, outfits, and character replacements, demonstrating strong zero-shot generalization.
  • W3 & Q2 (Visual Simulator for Robot Planning) — resolved in A3 (Response Table 3) and Appendix F.11 (Fig. 12): Used LiveAct to preview π0.5\pi_{0.5} candidate trajectories with VLM scoring for test-time selection, improving task success by +9.0% on five RoboTwin tasks and +56.2% (25.0% → 81.2%) on RoboFactory StackCube.

If you have any unresolved issues or new questions, please let us know. We are more than happy to answer any further questions you may have!

Thank you once again for your consideration.

Best regards,

Authors

Comments on the paper

1 comment
AuthorsOpenReview ↗

Global Response

First and foremost, we would like to express our sincere gratitude to the Reviewers, Area Chairs, and Program Chairs for their time and dedicated effort in reviewing our paper. We have provided detailed point-by-point responses under each Reviewer's section (Reviewer hMBV, Reviewer aJMV, Reviewer yXwe).

We are encouraged by the recognition of our work from the reviewers across three key dimensions:

  • Methods: "Explicitly separates the action recipient from the action's semantics and numerical values" (Reviewer yXwe). "Well-motivated and effective approach to multi-agent control across heterogeneous action spaces" (Reviewer aJMV). "The core idea is sensible: injecting semantic information to guide the model, both in terms of character identification and action understanding" (Reviewer hMBV).
  • Evaluation: "Strong performance across multiplayer games and multi-robot manipulation, with improvements in visual quality, action following, and agent attribution" (Reviewer aJMV). "Combines gaming and robotics dataset, while at the same time separating the two tasks in some evaluations" (Reviewer hMBV).
  • Presentation: "Clear and easy to follow" (Reviewer yXwe, Reviewer aJMV). "Easy to read and to follow" (Reviewer hMBV).

In response to the questions, we have conducted six sets of additional experiments and clarified our problem scope and method design in the point-by-point replies below, with corresponding qualitative results added to the revised PDF (highlighted in blue\textcolor{blue}{blue}). A summary of the PDF updates follows:

  • Appendix Fig. 8 (Appendix F.7)
    • To address the fair-comparison concern from Reviewer hMBV, we specify the exact checkpoints of all baselines and retrain MultiWorld, ActionParty, and a LingBot-style text-action baseline on our dataset using the same Wan2.2-TI2V-5B backbone. LiveAct outperforms all compared methods across both games and robotics.
  • Appendix Fig. 9 (Appendix F.8)
    • To address the questions on agent scalability and challenging interactions from Reviewer aJMV, we report quantitative action-following accuracy, attribution accuracy, and inference speed (FPS) for 2, 3, and 4 agents, and evaluate per-frame control and attribution under partial overlap, heavy occlusion, and visually similar agents.
  • Appendix Fig. 10 (Appendix F.9)
    • To address the question on 81-frame video length from Reviewer yXwe, we evaluate 241-frame streaming rollouts across consecutive 80-frame segments against retrained 5B baselines.
  • Appendix Fig. 11 (Appendix F.10)
    • To address the questions on generalization from Reviewer yXwe and Reviewer hMBV, we evaluate unseen control layouts, paraphrased agent descriptions, and first-frame visual edits with unseen backgrounds, outfits, and character replacements.
  • Appendix Fig. 12 (Appendix F.11)
    • To address the question on downstream robot planning from Reviewer yXwe, we follow WorldLine and use LiveAct to preview candidate trajectories for test-time planning on RoboTwin and RoboFactory StackCube, achieving substantial gains in task success.

We sincerely hope to engage in constructive dialogue with reviewers as we believe this exchange is vital for improving the quality of our work.

Best regards,

Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.