acceptodds
Withdrawn from ICLR 2027

OpenWorldAct: Open-Vocabulary Event Control for Real-Time Interactive World Models

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

We introduce **OpenWorldAct**, a real-time interactive world model that executes temporally grounded open-vocabulary events while achieving accurate navigation control from a third-person view. Most existing world models support only exploratory action spaces and do not allow users to control open-vocabulary events. The central obstacle is data: gameplay videos provide frame-synchronous controls but cover only a narrow range of semantics, while open or generated videos contain diverse events but can provide only inaccurate control signals obtained through estimation. Data that satisfy both requirements are difficult to obtain. To overcome this bottleneck, we build **ActMosaic**, a complementary data engine that divides its data into two streams. The gameplay stream retains frame-synchronous keyboard and mouse inputs to represent local navigation intent, while the generated event stream is organized around controllable characters, environments, and character–world interactions. At inference time, OpenWorldAct uses a unified time-aligned prompt interface to specify navigation controls and open-vocabulary events during the same rollout. We further develop a dual-resolution streaming few-step autoregressive model, where a low-resolution causal rollout is refined online by a high-resolution branch, enabling long-horizon real-time 720p generation. As an application showcase, we build a Gameplay Harness that maps a user's world setting to state-conditioned skills and tool-assisted observations. Comprehensive experiments show that OpenWorldAct substantially outperforms the evaluated systems in event control while achieving comparable wandering control and visual quality. **We will open source the model weights, code, data and benchmark**.

Primary area:
generative models
Keywords:
world model, video generation

Reviews

Among:
Shape:
1234this paper, Bottom 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.25 over 4 reviews, above 49% of the others.
Reviewer_bBNCOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

The paper proposes OpenWorldAct, an interactive video world model that combines precise, continuous navigation control with temporally grounded open-vocabulary event execution. The paper points out a main challenge in collecting training data that are both accurate in low-level navigation control and diverse in semantic event generation. To address this challenge, the paper proposes ActMosaic, a complementary data engine that combines framewise annotated gameplay recordings with motion-suppressed synthetic videos with temporal annotated event descriptions. These two data streams provide separate supervision for navigation / event, and both get transformed into a unified language space for training. Then a Gameplay Harness is leveraged for generating consistent world rules and events. The paper also designs a dual-resolution architecture for real-time inference, which achieves 25FPS on 2 GPU.

Critical strengths

Clear presentation and a well-motivated problem

  • The paper is easy to follow and the central problem is important and well-motivated. Since the controllability and customizability are key functionalities of world models. Navigation and Event Occurrence are both required for a game-like world model to be really usable.

Practical and scalable data strategy

  • The ActMosaic data collection strategy that avoid navigation and event occurrence appear at same time is scalable. Even if videos with high-quality annotations of both types become available in the future, this data can still be put to use.

Stronger empirical event-control performance

  • The method demonstrates much better event-response against different baselines.

An integrated real-time system

  • The proposed system has a high degree of completion, including controllable long video generation, dual-resolution real-time streaming deployment, and stateful VLM world interaction harness.
Critical weaknesses

1. Reliability of synthetic event supervision

  • The event-data construction pipeline firstly generates event descriptions with LLMs and then uses a video generation model to synthesize the videos. Although the paper mentions an event-adherence filter, it remains unclear how the method ensures the occurrence of the events and the accurate temporal alignment of these events.

2. Lack of real-time inference settings & results

  • The paper reports several setting for the real-time inference (denoising steps / resolutions / AR chunk size / AR history size / GPU of 80G 989 TFLOPS), while some settings (Quantization / Kernel Usage / Compilation Usage / TinyVAE Usage) are not reported. And DiT throughputs are reported, while end to end latency (including latency of input group waiting , DiT, upscaler, VAE and gameplay harness) and end to end fps are not reported. It is better to complete these information for real-time inference.

3. Missing analysis of event fidelity in the dual-resolution architecture

  • Table 2(c) compares 4-step generation at 360p, 480p, and 720p together with the 4+1 dual-resolution pipeline, showing a trade-off between efficiency and visual quality. However, the comparison does not include event-related metrics such as event completion, response rate, or temporal IoU. Could the authors report event-related metrics across different resolution, especially the direct 720p generation one?

4. Joint control composition remains imperfect

  • The performance still decreases a lot when both navigation and event prompts are active. Could the authors report more details on these cases? (e.g. different camera & movement speed / different event types / multiple event at same time)
Questions for authors

See weaknesses.

Reviewer_7BcBOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

OpenWorldAct combines keyboard/mouse navigation with temporally controlled, open-vocabulary events. It uses separately supervised gameplay and motion-suppressed event videos, time-aligned textual conditioning, and dual-resolution streaming generation.

Critical strengths
  • The problem is practically important, and learning joint execution without paired control–event demonstrations is a useful contribution.
  • Event-execution gains are substantial, with supporting results on a public WBench subset and in human evaluation.
  • The appendix provides meaningful evidence: a filter audit, semantic and compositional breakdowns, and a matched composition diagnostic that explicitly acknowledges interference
Critical weaknesses
  1. The incremental contribution needs stronger substantiation. Related work such as ActWorld already combines navigation and interaction, textual control conditioning, and multi-resolution few-step generation. The distinctive value appears concentrated in motion-suppressed complementary supervision and the specific streaming design, whose incremental benefits need more targeted validation. This concerns contribution strength, not missing comparisons with concurrent work.
  2. Evaluation reliability remains insufficiently established. On 67 WBench cases, the overall score improves from 73.54 to 74.67 without reported uncertainty estimates (Table 8). Only 65 of 198 human comparisons involve OpenWorldAct, including 14 against LingBot-World-V2: 7 wins, 5 ties, and 2 losses (Table 11). Although Appendices C.2 and E document scoring rubrics and human ratings, exact event/control judge versions, complete prompts, and human agreement with automated event scores and temporal boundaries remain unreported. These gaps limit reproducibility and confidence in the magnitude of the advantages.
  3. The role of prompt enhancement is insufficiently specified and isolated. Section 4.1 describes decomposing complex instructions into atomic events, while Appendix C.3 details prompt assembly and temporal injection. However, the decomposition procedure is not described in reproducible detail, no dedicated raw-versus-enhanced prompt ablation is reported, and the enhancer’s application across evaluated systems is not explicitly specified. These gaps leave its contribution to the reported gains uncertain.
Questions for authors
  1. Novelty and contribution: What are the main technical distinctions from closely related navigation-and-interaction frameworks such as ActWorld? Which results most directly substantiate the incremental benefits of complementary supervision and the proposed streaming design?
  2. Evaluation reliability: Which model versions and prompts are used for event/control evaluation, and how well do their judgments agree with human annotations? Can the authors report uncertainty estimates for the WBench results and the human preference comparison against LingBot-World-V2, particularly given the limited direct comparisons?
  3. Prompt enhancement: How is the enhancer implemented, and how is it applied across the evaluated systems? Is there a same-model comparison of raw versus enhanced instructions that isolates its contribution to event-execution performance?
Reviewer_ht1TOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper presents OpenWorldAct, a real-time interactive world model that supports both navigation and open-vocabulary event control. It introduces ActMosaic, which combines gameplay recordings with motion-suppressed synthetic event data, and a dual-resolution autoregressive framework for real-time generation. Experiments demonstrate improved event control while maintaining competitive navigation accuracy and visual quality.

Critical strengths

S1. Strong empirical performance on open-vocabulary event control.

  • Table 1 demonstrates substantial improvements over the evaluated baselines, increasing average event completion from 0.947 to 1.957 (on a 0–3 scale) and temporal IoU from 28.34% to 51.13%.
  • The improvements are not limited to simple events. Appendix D, Table 9 reports consistent gains across environment changes, character actions, coupled action–effect processes, and target interactions.
  • The evaluation includes both automatic metrics and a blinded human study, providing complementary evidence for the reported improvements.

S2. Practical data construction strategy for complementary supervision.

  • ActMosaic (Section 3, Figure 2) addresses a relevant data limitation: gameplay provides accurately synchronized movement controls but limited event diversity, whereas generated videos can represent more diverse semantic events.
  • The separation into navigation and event supervision avoids requiring paired demonstrations for every possible navigation–event combination.
  • The motion-suppression strategy is supported by Table 2(a), where filtering partially recovers navigation accuracy while maintaining most of the event-control improvements introduced by synthetic event data.
Critical weaknesses

W1. Limited evaluation of navigation-dependent event interactions.

  • The model learns navigation and event control from separate data streams and combines them at inference time. However, the evaluation primarily uses randomly paired navigation trajectories and events, without sufficiently testing scenarios where event execution depends on navigation, such as approaching an object before interacting with it.
  • Moreover, Table 10 shows that adding navigation reduces Event Temporal IoU from 72.95% to 56.22%. These results raise questions about whether the model can handle spatially and causally dependent interactions, beyond simply executing navigation and events simultaneously.

W2. Insufficient evaluation of stateful interactions through the Gameplay Harness.

  • The Gameplay Harness (Section 4.3) maintains explicit world states and updates skills and event prompts accordingly. However, Appendix B mainly provides qualitative examples without quantitatively evaluating state tracking or state transitions.
  • Furthermore, it remains unclear whether event-induced changes persist after camera movement or occlusion. More systematic evaluation is needed to establish whether the Gameplay Harness reliably supports stateful interactions.
Questions for authors

Q1. Joint navigation–event interference.

Appendix D.4 (Table 10) reports a noticeable decrease in event performance when navigation is introduced. How does this degradation vary with navigation complexity, camera motion, and spatially grounded character-object interactions?

Q2. Effectiveness of the Adaptive Gameplay Harness.

Could the authors provide quantitative evaluations of the Gameplay Harness, particularly regarding state tracking accuracy, state-dependent skill selection, and successful execution of state transitions?

Q3. Effect of the final distillation pipeline.

The ActMosaic ablations in Table 2(a) are conducted using a full-step AR proxy rather than the final distilled model. Do the observed improvements in navigation accuracy from motion filtering persist after the complete distillation pipeline?

Q4. Qualitative video demonstrations.

Could the authors provide supplementary videos showing complete, uninterrupted rollouts, particularly for simultaneous navigation and event control, navigation-dependent interactions, and long-horizon state changes? Since Figures 1 and 4 primarily present sampled frames, video demonstrations would help assess temporal consistency, event timing, and navigation responsiveness.

Reviewer_HYmBOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper presents OpenWorldAct, a real time world model that combines keyboard/mouse navigation with open vocabulary events such as character actions and environmental changes. The authors introduce ActMosaic, a dataset combining 119K gameplay clips with synchronized control signals and 4.7K generated event clips. Both types of supervision are represented as text and combined at inference time. The model uses block autoregressive generation and a dual resolution design. The authors also introduce a Gameplay Harness that translates natural language world descriptions into interactive state machines. On a 120 case benchmark covering 300 events, OpenWorldAct roughly doubles event completion over the strongest baseline while remaining competitive on navigation and visual quality. A user study also favors the proposed approach

Critical strengths
  • Supporting free form events during live rollouts is a useful step beyond navigation only world models
  • Separating control and event supervision is a simple approach that avoids the need for paired data
  • The gains are consistent: 72% vs 41% event response rate, the highest score on the WBench subset (74.67 vs 73.54), and a 90% win rate in the user study.
  • The D.4 analysis is useful and transparent. Adding navigation reduces event response from 90.6% to 78.3%, but the quality of successfully completed events remains essentially unchanged
Critical weaknesses
  • It is difficult to tell how much of the improvement comes from the proposed method versus the LTX 2.3 backbone and Happy Horse 1.1 event teacher.. removing event data drops completion to 0.81 but this mainly shows that event supervision helps. So how does the original - LTX 2.3 perform when prompted with events? would a baseline trained on ActMosaic achieve similar results?
  • The benchmark is largely constructed by the authors and event completion is evaluated using a VLM that is neither identified nor validated against human judgments
  • Some baselines required custom interfaces to support chunk level prompting and since this is central to the capability being evaluated, the authors should explain exactly what these adaptations involve
  • The seen/unseen split only measures overlap with the event training captions and does not account for the teacher's/backbone's pretraining data. Testing on an entirely held out event category would provide stronger evidence of open vocabulary generalization
  • I find the navigation performance to be less convincing than the paper suggests. Camera accuracy is lower than LingBot's (74.2 vs 88.3) and visual quality is below Alaya Evoke (79.8 vs 82.1), and then human ratings favor OpenWorldAct's navigation over LingBot's (4.25 vs 3.79)? I believe this discrepancy deserves some explanation
  • The user study is relatively small (with 13 raters and 198 judgments), some system pairs have only nine judgments, and no confidence intervals are reported
  • The Gameplay Harness is demonstrated through only two examples while long horizon behavior is evaluated over just 30 seconds.. the paper also provides limited detail on gameplay data sources and total training compute
Questions for authors

See above

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.