acceptodds
Under review as a conference paper at ICLR 2027

Learning to Act and Edit in Video Worlds

94 views

Abstract

Interactive video world models have made substantial progress in generating explorable environments with navigation control. However, extending such control to diverse character actions and environmental events is hindered by fragmented supervision across heterogeneous video sources. To bridge this gap, we develop a unified data pipeline that transforms real-world videos, gameplay video, edited videos and synthetic videos into a temporally aligned dataset for camera control, character actions, and environmental events. The resulting dataset contains 180,162 clips, covering 91 action types across 14 visual domains. Using this dataset, we train an autoregressive world model initialized from a pretrained bidirectional video generator . Experiments show that the resulting model (i) achieves strong instruction following across everyday interactions, gameplay actions, visual effects, and scene-level edits while preserving spatial navigation fidelity, (ii) exhibits zero-shot compositional generalization across unseen instruction combinations alongside precise camera-actor disentanglement, and (iii) reveals the distinct roles of different data sources, yielding a reproducible, resource-efficient recipe for training semantically controllable video world models. We will release the code and dataset.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Interactive world model, Video Data Curation

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_n93vOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

This paper presents a data pipeline and autoregressive video world model for control over camera motion, character actions, and environmental events. The pipeline aggregates real-world, gameplay, edited, and synthetic videos into a temporally aligned dataset of 180,162 clips across 91 action types and 14 visual domains, then fine-tunes a Wan2.2 generator with chunk-aligned conditioning and causal rollout. The authors report instruction-following, navigation-preservation, compositional generalization, and cross-source results, together with data-composition ablations.

Critical strengths
  1. Unifying camera movement, character actions, and scene edits under a shared temporally aligned control interface is a useful research problem. The dataset pipeline addresses fragmented supervision and the source-composition analyses provide practical insight into which data types support which capabilities.
  2. The study evaluates several action and event categories and includes ablations of data sources, causal training, and compositional instruction combinations. This breadth is valuable for understanding the capabilities and trade-offs of controllable video world models.
Critical weaknesses
  1. The paper's principal contribution is a curated dataset and training recipe on top of a large pretrained video model, while the methodological novelty over existing interactive world models appears limited. The large dataset is dominated by gameplay recordings, and the reported domain/action breadth does not by itself establish that control generalizes beyond the source distributions. Stronger held-out-game/source and genuinely out-of-combination tests are needed to substantiate broad compositional claims.
  2. The main evidence emphasizes model-scored instruction adherence and benchmark scores. The paper does not sufficiently demonstrate that generated videos preserve causal consistency or reliable long-horizon controllability under sequential interventions, which are central properties of a world model. It also needs tighter comparisons that control for backbone and training data scale. These limitations weaken the claim that the proposed recipe constitutes a reproducible, resource-efficient advance rather than a substantial task-specific fine-tuning effort.
Questions for authors
  1. Are train/test splits disjoint at the source-video, game, and scene level? Please report performance on unseen games or source datasets and on action/event compositions not represented in training, since this would materially affect the generalization claim.
  2. What are the parameter-matched and compute-matched comparisons to current interactive video world models, and how is causal consistency measured during multi-step closed-loop rollouts rather than short instruction-following clips?
Reviewer_EU48OpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper presents Cela-World-180K, a dataset of 180,162 clips covering 91 action categories and 14 visual domains, together with a 5B interactive video model. The data pipeline combines real-world, gameplay and synthetic videos, refines action boundaries, and produces temporally aligned descriptions of scenes, actions and events. The model builds on Wan2.2 TI2V-5B with camera conditioning and chunk-level text control, followed by causal training and four-step distillation. It improves the WBench Navi average from 69.1 to 76.9 and the Full-split average from 61.1 to 69.8. Qualitative examples cover tool changes, environmental edits and combinations of actions and scene changes.

Critical strengths
  1. The dataset brings together a useful range of supervision that is usually scattered across separate sources. Gameplay supplies control-aligned interactions, real videos supply appearance and camera diversity, and synthetic clips add events that are harder to collect. The resulting coverage of 91 actions, 50 games and multiple viewpoints makes this a worthwhile resource for studying more than navigation alone.

  2. The temporal annotation pipeline addresses a concrete problem in interactive video training. Coarse input logs are refined using visual action boundaries, clips retain context around the action, and the resulting captions are aligned with generation chunks. Connecting these labels to separate camera and semantic controls gives the model a usable interface for changing both viewpoint and scene content.

  3. The WBench results show consistent gains over the base model across all five reported dimensions on both splits (Table 1). The Full-split interaction score rises from 28.3 to 49.2, while consistency also improves. The qualitative examples give a helpful picture of the broader action and editing behavior behind those aggregate scores.

Critical weaknesses
  1. The paper does not yet establish which parts of the data recipe drive the gains. Section 5.2 reports a navigation warm-up comparison, but the claimed analysis of source composition and annotation granularity lacks matched-budget source-removal and global-versus-chunk-caption results. The comparison with Wan2.2 changes the data, conditioning and training stages together, making the proposed recipe difficult to assess component by component.

  2. The evidence for compositional generalization is mostly qualitative, and the evaluation setup is unclear. Tool transfer uses ten synthetic examples and the composition study uses 500 post-training samples, but the paper does not specify which action-object-scene combinations are held out or report success rates across a defined test set. I could not tell how much of the demonstrated behavior comes from adaptation and how much transfers to unseen combinations.

  3. The annotation process needs more detail and an accuracy audit. The paper describes VLM-based localization, caption checks and deduplication without giving the model checkpoints, prompts, acceptance thresholds or measured temporal and semantic error rates. Since temporally accurate supervision is central to the dataset contribution, these details would help readers judge its quality and reproduce the collection process.

Questions for authors
  1. With the model and training budget fixed, how much do chunk-level captions and each action/editing source contribute to WBench performance? How do these effects compare with the reported navigation warm-up gain?

  2. What combinations are held out in the ten-example and 500-example studies, and what are the success rates over the full evaluation sets? Please clarify what is zero-shot at each stage of adaptation.

  3. Which VLMs, prompts and thresholds are used for action localization and caption verification? Can you report an annotation audit and explain how training clips are checked against evaluation sources for overlap?

Reviewer_aXUPOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper introduces Cela-World, a unified video world model designed to generate camera movements, character actions, and environmental and object changes. The authors construct Cela-World-180K, a dataset containing approximately 180,000 video clips from gaming, real-world, and synthetic sources, with fine-grained action supervision achieved through action localization, temporal alignment, and multi-level annotations. Based on this dataset, they develop Cela-World-5B using Wan2.2 TI2V-5B, adopting a multi-stage training strategy that combines camera-conditioned control, chunk-wise autoregressive generation, and causal distillation to improve instruction following and temporal consistency. Experimental results show that Cela-World-5B outperforms the Wan2.2 baseline on benchmarks such as WBench and demonstrates promising cross-scene action transfer and compositional generalization.

Critical strengths
  1. This paper introduces the Cela-World-180K dataset, which covers the categories of Action, Navigation, and Editing. It serves as a valuable resource for researchers looking to train interaction world models, enabling them to get started quickly.

  2. This paper also invests significant effort in the data curation pipeline. Through multi-level captioning and a rigorous dataset curation and verification process, the authors describe each clip from various perspectives and filter the data across multiple dimensions. This results in a highly accurate dataset of 180,000 entries, which facilitates better model training while effectively preserving the model's original, general-purpose capabilities.

  3. This paper is logically coherent, clear, and easy to understand. It also features a wealth of visualizations that allow for a more direct view of the final results.

Critical weaknesses
  1. Although this paper curates and cleans 180,000 high-quality interactive data samples, it lacks sufficient innovation. The data pipeline follows a process similar to existing methods for cleaning video data, with only minor adjustments to specific details. Furthermore, the model training and validation phases do not constitute a significant innovation, and the empirical data insights provided are quite limited.

  2. As shown in Table 1 of the experiment section, training with Cela-World180k yields some improvement; however, the gap compared to existing baselines is not significant. For instance, Cela-World-5B outperforms DreamX-World by only 1.9 percentage points—a rather modest gain. Furthermore, the experimental analysis fails to address certain critical dimensions where performance is suboptimal; for example, why does the model underperform in the "physical" dimension?

  3. Why are the capabilities demonstrated in version 5.2 attributed to training on Cela-World180k? Could they instead be inherent capabilities of the Wan model itself, or perhaps the result of well-crafted prompts that elicited such outputs? Is it possible to provide comparative visualizations—do other models exhibit similar capabilities? Furthermore, could these empirical data insights be further substantiated through quantitative experiments?

Questions for authors
  1. Could you provide more details on how you used MiniMax H3 to construct your synthesized video?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.