- Rating
- 2
- Confidence
- 4
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
The paper presents AlayaWorld, an autoregressive video world model that supports camera-controlled exploration and text-driven events during continuous generation. It builds a persistent 3D point cache from generated observations and renders it at requested viewpoints, allowing previously visited scenes to be recalled beyond the recent context window. Compressed temporal history and recent-motion conditions help maintain continuity between chunks, while text instructions can change without resetting the visual history. The authors also bring six real-world and gameplay datasets under a shared annotation protocol and introduce GenEvent, a synthetic dataset for behaviors such as spell casting and summoning. Training with corrupted histories and replayed prediction errors helps the model handle imperfect context, followed by distillation to four-step sampling. The main contribution is the integration of spatial memory, temporal conditions, and event supervision into a playable video generation framework. Results on iWorld-Bench, WorldMark, and WBench show improvements in scene consistency and revisitation memory, with qualitative examples demonstrating continued exploration and changing events.
Critical strengths
-
The paper is clearly written. It explains the roles of spatial memory, temporal history, and recent-motion conditions, and how they are used during continuous generation. The overall workflow is well organized and easy to follow.
-
The visual examples look good, particularly those showing scene revisits, continued exploration, and text-driven events. The memory and consistency results on several benchmarks provide quantitative support for these examples and make the system’s practical value convincing.
Critical weaknesses
-
The new technical contribution is not clear enough to me, and I do not yet see a key methodological improvement over prior work. The contribution seems stronger on the engineering side than in algorithmic novelty. Building video world models does require integrating several techniques, though, and I value the work involved in making them function together. I still view the paper positively.
-
The training settings are incomplete for reproduction. The paper explains the training stages, losses, and some parameter-freezing choices, but does not provide the concrete configurations or clearly state plans to release the code, models, and data. Could the authors clarify these plans? For work whose main contribution is system integration, access to these resources is important to its value for the community. Without such a release, I would expect the impact of the work to be more limited.
Questions for authors
Please refer to the two questions in the Weaknesses section.
2 comments
Thank you for the detailed response and additional experiments. I will maintain my original rating.
Response
We thank the reviewer for the constructive comments and valuable suggestions, which have helped us better articulate our methodological contributions and provide clearer information for reproducibility.
Response: Thank you for recognizing the value of this work. Our methodological contribution is a unified conditioning design that connects persistent scene recall, recent dynamics, and camera control during chunk-wise autoregressive generation.
Spatiotemporal memory and camera control. The requested camera trajectory determines how the persistent 3D cache is rendered, so the spatial condition provides both previously observed scene content and the desired viewpoint. Compressed temporal history supplies recent appearance and dynamics, while the appearance anchor and recent-motion condition support continuity across chunks. This design connects long-term spatial recall with the temporal context needed to continue an evolving scene.
Consistent conditioning across training and inference. A key design choice is to construct these visual conditions consistently with the target video's causal-VAE representation. Spatial renderings are encoded as a sequence with a causal prefix, temporal-history windows are aligned in pixel space before encoding, and chunk handoff follows the same encoding convention during training and inference. These choices keep the temporal structure of the conditioning signals aligned with the generated video throughout the autoregressive rollout.
Training pipeline. A further distinction is that we do not train a separate camera-control branch in Stage 1; when training on heterogeneous data, the resulting first-stage model substantially eases the final few-step distillation in Stage 3.
Instruction-driven events. We also introduce GenEvent to provide supervision for events triggered by changing text instructions, supporting interaction beyond camera navigation.
We will revise the contribution statement and method discussion to explain these design choices and their distinction from existing components more clearly.
Response: We provide the main training hyperparameters and training budgets below.
Training configurations. All stages use AdamW, a weight decay of 0.001, a per-GPU batch size of 1, and random seed 42. No gradient accumulation is used, so the global batch size equals the number of GPUs. Our training comprises four stages: Stage 1 performs bidirectional video pretraining, Stage 2a pretrains the history encoder, Stage 2b performs autoregressive fine-tuning with temporal and spatial conditioning, and Stage 3 distills the autoregressive teacher into a four-step student.
Stages 1 and 2b use full Transformer fine-tuning, while Stages 2a and 3 use LoRA. The learning rate remains constant after linear warm-up. In Stage 3, the listed learning rate and warm-up apply to the student optimizer; the critic learning rate is 5e-5, with a 10:1 critic-to-student update ratio and a consistency-regularization weight of 0.7.
Stage 3 uses a two-segment rollout curriculum: approximately 1,000 global steps with up to five chunks, followed by 1,500 global steps with up to ten chunks.
Release commitment. We commit to publicly releasing the training code and model checkpoints for all stages, together with the inference code. The release will include the corresponding configuration files, environment setup, and instructions for training and inference to support reproduction and further research.