acceptodds
Desk rejected at ICLR 2027

AlayaWorld: Interactive Long-Horizon and Playable Video World Generation

Desk rejected at ICLR 2027, so left out of the paper lists and the map.

Abstract

Building playable virtual worlds traditionally requires substantial manual effort in asset creation and interaction design. Video world models offer a generative alternative, but sustained interaction requires models to respond to user controls and perform diverse actions while maintaining long-horizon scene consistency and generation stability. We present AlayaWorld, an open video world model for long-horizon interactive and playable world generation, supporting camera-controlled navigation and text-driven behaviors such as spell casting during continuous generation. AlayaWorld generates video autoregressively in chunks, coordinating persistent spatial memory, temporal history, and motion-aware conditions to support scene recall across viewpoints and continuous evolution across chunks. To support navigation and interactive behaviors across diverse scenes, we curate and annotate six real-world and gameplay datasets under a unified protocol and further construct GenEvent, a synthetic video dataset that supplements this mixture with supervision for instruction-driven behaviors such as spell casting. We combine training on imperfect histories with autoregressive distillation to improve long-horizon generation stability and enable four-step sampling per chunk. Experiments on iWorld-Bench, WorldMark, and WBench demonstrate the effectiveness of AlayaWorld in world consistency, generation stability, and interactive control. We provide an open research framework with reference implementations and reproducible pipelines for interactive and playable video worlds.

Primary area:
generative models
Keywords:
world model, video generation

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_YhKTOpenReview ↗
Rating
2
Confidence
4
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

The paper presents AlayaWorld, an autoregressive video world model that supports camera-controlled exploration and text-driven events during continuous generation. It builds a persistent 3D point cache from generated observations and renders it at requested viewpoints, allowing previously visited scenes to be recalled beyond the recent context window. Compressed temporal history and recent-motion conditions help maintain continuity between chunks, while text instructions can change without resetting the visual history. The authors also bring six real-world and gameplay datasets under a shared annotation protocol and introduce GenEvent, a synthetic dataset for behaviors such as spell casting and summoning. Training with corrupted histories and replayed prediction errors helps the model handle imperfect context, followed by distillation to four-step sampling. The main contribution is the integration of spatial memory, temporal conditions, and event supervision into a playable video generation framework. Results on iWorld-Bench, WorldMark, and WBench show improvements in scene consistency and revisitation memory, with qualitative examples demonstrating continued exploration and changing events.

Critical strengths
  1. The paper is clearly written. It explains the roles of spatial memory, temporal history, and recent-motion conditions, and how they are used during continuous generation. The overall workflow is well organized and easy to follow.

  2. The visual examples look good, particularly those showing scene revisits, continued exploration, and text-driven events. The memory and consistency results on several benchmarks provide quantitative support for these examples and make the system’s practical value convincing.

Critical weaknesses
  1. The new technical contribution is not clear enough to me, and I do not yet see a key methodological improvement over prior work. The contribution seems stronger on the engineering side than in algorithmic novelty. Building video world models does require integrating several techniques, though, and I value the work involved in making them function together. I still view the paper positively.

  2. The training settings are incomplete for reproduction. The paper explains the training stages, losses, and some parameter-freezing choices, but does not provide the concrete configurations or clearly state plans to release the code, models, and data. Could the authors clarify these plans? For work whose main contribution is system integration, access to these resources is important to its value for the community. Without such a release, I would expect the impact of the work to be more limited.

Questions for authors

Please refer to the two questions in the Weaknesses section.

2 comments

We thank the reviewer for the constructive comments and valuable suggestions, which have helped us better articulate our methodological contributions and provide clearer information for reproducibility.

W1: The methodological contribution and its distinction from existing techniques need to be stated more clearly.

Response: Thank you for recognizing the value of this work. Our methodological contribution is a unified conditioning design that connects persistent scene recall, recent dynamics, and camera control during chunk-wise autoregressive generation.

Spatiotemporal memory and camera control. The requested camera trajectory determines how the persistent 3D cache is rendered, so the spatial condition provides both previously observed scene content and the desired viewpoint. Compressed temporal history supplies recent appearance and dynamics, while the appearance anchor and recent-motion condition support continuity across chunks. This design connects long-term spatial recall with the temporal context needed to continue an evolving scene.

Consistent conditioning across training and inference. A key design choice is to construct these visual conditions consistently with the target video's causal-VAE representation. Spatial renderings are encoded as a sequence with a causal prefix, temporal-history windows are aligned in pixel space before encoding, and chunk handoff follows the same encoding convention during training and inference. These choices keep the temporal structure of the conditioning signals aligned with the generated video throughout the autoregressive rollout.

Training pipeline. A further distinction is that we do not train a separate camera-control branch in Stage 1; when training on heterogeneous data, the resulting first-stage model substantially eases the final few-step distillation in Stage 3.

Instruction-driven events. We also introduce GenEvent to provide supervision for events triggered by changing text instructions, supporting interaction beyond camera navigation.

We will revise the contribution statement and method discussion to explain these design choices and their distinction from existing components more clearly.


W2: Reproducibility requires concrete training configurations and an explicit commitment to release the code and model checkpoints.

Response: We provide the main training hyperparameters and training budgets below.

Training configurations. All stages use AdamW, a weight decay of 0.001, a per-GPU batch size of 1, and random seed 42. No gradient accumulation is used, so the global batch size equals the number of GPUs. Our training comprises four stages: Stage 1 performs bidirectional video pretraining, Stage 2a pretrains the history encoder, Stage 2b performs autoregressive fine-tuning with temporal and spatial conditioning, and Stage 3 distills the autoregressive teacher into a four-step student.

ConfigurationStage 1Stage 2aStage 2bStage 3
Learning rate1e-55e-55e-51e-5
Training steps (approx.)12,0006,00015,0002,500
Number of GPUs641122424
Global batch size641122424
LR warm-up (optimizer updates)200100300100
Gradient clipping (max norm)1.05.05.05.0
LoRA rank / alpha—256 / 256—256 / 256

Stages 1 and 2b use full Transformer fine-tuning, while Stages 2a and 3 use LoRA. The learning rate remains constant after linear warm-up. In Stage 3, the listed learning rate and warm-up apply to the student optimizer; the critic learning rate is 5e-5, with a 10:1 critic-to-student update ratio and a consistency-regularization weight of 0.7.

Stage 3 uses a two-segment rollout curriculum: approximately 1,000 global steps with up to five chunks, followed by 1,500 global steps with up to ten chunks.

Release commitment. We commit to publicly releasing the training code and model checkpoints for all stages, together with the inference code. The release will include the corresponding configuration files, environment setup, and instructions for training and inference to support reproduction and further research.

Thank you for the detailed response and additional experiments. I will maintain my original rating.

Reviewer_vvpyOpenReview ↗
Rating
3
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

AlayaWorld is a chunk-wise autoregressive video world model built on a pretrained video diffusion model, LTX-2.3. To maintain the long-horizon memory, several complementary conditions are injected together: 1) a persistent appearance anchor, denoted as sink tokens; 2) a learned, compressed 2D temporal history from a 25-frame window; 3) a persistent 3D point cache estimated by ViGeo, rendered along the requested camera trajectory to give target-view spatial evidence; 4) a recent-motion latent at the chunk boundary. Meanwhile, the text instructions can be changed at chunk boundaries to trigger events such as spell casting. Evaluating on various benchmarks, the proposed approach achieves the best overall scores among most compared systems. Overall, the paper is well-written and solving an important problem in world models.

Critical strengths
  1. A hybrid memory design that works well in practice. The paper combines a learned compressed 2D latent history, which captures recent appearance and dynamics, with a rendered 3D point-cloud cache, which gives viewpoint-dependent recall of regions that have left the temporal window. Sink tokens and a recent temporal condition support this further. Each pathway covers a different failure mode: 2D history alone forgets regions after long detours, and 3D rendering alone cannot synthesize newly exposed or dynamic content.
  2. Comprehensive experiments with strong results. The method is evaluated on three complementary benchmarks against a broad set of recent world models. Moreover, the recipe and data pipeline are provided in detail, which should make the work useful to the community.
Critical weaknesses
  1. The fusion of the 2D compressed history and the 3D spatial rendering is under-analyzed, especially when one condition has clear artifacts. a) No explicit arbitration. Both conditions are concatenated with the anchor, the motion latent and the noisy target, and handled by full attention. Invalid rendered regions are removed only through a binary validity mask; no mechanism decides between the pathways when they disagree. b) Artifacts are likely in practice. The point cache is built from ViGeo estimates on generated frames, after a heuristic scale alignment (median of pairwise displacement ratios) and with intrinsics fixed per clip. So floaters, misregistration, scale drift and ghosting of dynamic objects are to be expected. Meanwhile, the compressed history may carry accumulated autoregressive drift. c) Robustness training does not cover the relevant failure modes. The Helios-style corruption (Gaussian noise, blur, saturation) and the error replay perturb the conditions independently. They do not simulate inconsistent evidence, i.e., geometric misregistration in the rendering that contradicts the history. d) The paper's own ablation shows sensitivity. Replacing ViGeo with DA3 lowers Video Quality from 79.3 to 73.9 and Interaction from 80.0 to 72.2 (Table 4). This suggests the generator relies strongly on the rendered evidence, and that rendering errors can propagate to the output instead of being corrected by the temporal pathway. Robust memory under imperfect geometry is the central claim. Without an analysis of conflicting conditions, it is unclear how reliable the hybrid memory is beyond the benchmark scenes.
  2. The evaluation of memory and consistency is incomplete and relies mainly on 2D metrics. a) Mostly 2D measurements. The paper focuses on pixel-level consistency along revisit trajectories: iWorld-Bench and WorldMark. Nothing directly tests whether the generated world is 3D-consistent: multi-view geometric coherence of the whole rollout, agreement between the point cloud at first visit and at revisit. Maybe Met3R can be used to measure the revisit 3D consistency. b) Incomplete WBench results. Only the Consistency group is reported, although WBench also defines Video Quality, Setting, Interaction and Physical dimensions. A full table is needed to support the claims about interactive control. c) The ablation lacks memory metrics. Table 4 reports only WBench group aggregates. It does not use the memory-specific metrics (WorldMark LM/GM/RM, iWorld-Bench memory symmetry and trajectory alignment, return-view similarity). So it does not show how much each module contributes to revisit memory specifically, which is the paper's main claim. d) The "w/o spatial memory" variant is confounded. It also changes camera control from trajectory renderings to text prompts, which mixes the memory effect with the control effect.
Questions for authors

Clarifications that could change my assessment:

  1. The framing is "interactive" and "playable". Please report wall-clock time per chunk (DiT, VAE, and ViGeo + rendering, separately), FPS and peak GPU memory as a function of rollout length, for the 4-step student.
  2. Please add a quantitative evaluation of text-driven events (e.g., VLM/human-judged success rate and scene preservation after an instruction switch) and a GenEvent on/off ablation.
3 comments

We appreciate the reviewer’s careful assessment and thoughtful suggestions, which have been valuable in improving the rigor of our analysis and the completeness of our evaluation.

W1: The fusion of temporal history and spatial rendering requires further analysis when the two conditions contain conflicting or inaccurate evidence.

Response: The temporal and spatial conditions are fused through learned attention, while the validity mask removes invalid rendered regions. We do not use a separate arbitration module. Our corruption training exposes the model to imperfect conditions, but geometric disagreement between the two pathways requires a more targeted analysis.

To examine this issue, we conducted additional experiments on eight scenes, keeping the requested camera controls and recent history fixed while introducing yaw errors into the spatial rendering.

Rendering yaw errorDINO return similarityDA3 geometry score
0°0.80360.8643
5°0.78110.8137
10°0.71310.8188

Return similarity decreases as the yaw error increases, and both perturbed conditions have lower geometry scores than the unperturbed reference. These results directly characterize the sensitivity to spatial misregistration. The ViGeo/DA3 ablation likewise shows that the quality of geometric conditioning affects performance.

Dynamic-object ghosting. Masking dynamic objects and reconstructing only the static background can mitigate this issue. For WBench, we use the benchmark-provided subject masks, dilated by 12 pixels, to exclude foreground regions from the static geometry cache. This prevents the masked foreground regions from being stored as persistent scene geometry and helps reduce ghosting and duplicated structures on revisits.

Short-lived spell effects, such as flashes or smoke, may appear outside the provided subject masks. DA3 and ViGeo also struggle to reconstruct scenes after spell-casting events, which contributes to the low scene-preservation scores.

W2: Memory and consistency evaluation requires direct 3D measurements, complete WBench results, and memory-specific ablations that account for changes in camera conditioning.

Response:

3D consistency evaluation. We conducted additional evaluation on 50 test samples, measuring MEt3R and point-cloud Anti-Hallucination (AH) between the first visit and revisit. These measurements complement the original image-based consistency metrics.

MetricResult
MEt3R ↓0.141
AH ↑78.25

AH computation. We adopt the Anti-Hallucination metric from WorldRoamBench [1]. Given aligned first-visit and revisit point clouds, let dcrossd_{\mathrm{cross}} denote the mean nearest-neighbor distance from the revisit cloud to the first-visit cloud, and dwithind_{\mathrm{within}} the mean nearest-neighbor distance between two disjoint random subsets of the revisit cloud. AH is reported on a 0–100 scale:

AH=100×min⁡(1,dwithindcross).\mathrm{AH}=100\times\min\left(1,\frac{d_{\mathrm{within}}}{d_{\mathrm{cross}}}\right).

Higher AH scores indicate that the geometry reconstructed upon revisit is better supported by the first-visit observations.

Complete WBench results. We provide the full results on the WBench navigation split (158 navigation cases) below, covering all five groups: Video Quality, Setting, Interaction, Consistency, and Physical. All scores range from 0 to 100, with higher values indicating better performance. Bold indicates the best result in each row.

MetricHY-World 1.5HY-GameCraftLingBot-FastLingBot-v2AlayaWorld
Video Quality
Aesthetic60.152.662.664.462.8
Imaging65.458.763.867.567.7
Flickering93.593.792.491.492.7
Dynamic91.196.895.696.291.1
Smoothness98.197.696.096.597.0
HPSv3-Norm60.538.365.774.664.3
Average78.173.079.481.879.3
Setting
Scene53.550.663.466.751.6
Subject90.882.592.486.987.7
Average72.266.677.976.869.7
Interaction
Navigation87.567.879.482.880.0
Consistency
Background92.786.590.992.594.1
Spatial90.660.577.282.387.9
Gated Spatial84.960.576.978.781.9
Segment100.099.498.198.198.1
Perspective62.517.982.884.586.6
Subject89.182.688.688.993.4
Geometric92.088.385.487.194.1
Photometric83.185.079.179.880.3
Average86.972.684.986.589.5
Physical
Causal Fidelity74.068.372.576.765.1
Visual Plausibility58.656.558.861.461.1

AlayaWorld achieves the highest Consistency average (89.5). We acknowledge that performance on the other dimensions still leaves room for improvement. We attribute part of this gap to the relatively limited representation of WBench-like third-person data in our training set. This limitation can become more noticeable in Stage 3, where few-step distillation may introduce an additional drop in visual quality relative to earlier stages. In our experience, even a small amount of additional fine-tuning on training data with a similar format or domain can noticeably improve performance on these cases.

Memory-specific evaluation and camera conditioning. As stated in the paper, the variant without spatial memory uses text prompts for camera control. Our training includes conditioning dropout to support this text-based camera-control mode. We nevertheless agree that this ablation removes both historical spatial information and trajectory-conditioned rendering, so the reported drops cannot be attributed entirely to persistent memory.

To further examine the role of historical information, we conducted additional experiments on eight scenes, keeping the requested camera trajectories and rendering interface. We either restricted retrieval to the most recent 32 RGB frames or blocked access to pre-reset observations from chunk 27 onward.

ConditionDINO return similarityDA3 geometry scoreEstimated translation alignment
Full reference0.80360.86430.6808
Recent-only retrieval0.69580.75040.6039
Retrieval reset0.69400.78170.4583

Both restrictions reduce performance across all three metrics, supporting the benefit of retrieving older observations. These return-based measurements complement the aggregate WBench scores by directly examining access to historical information.

We agree that a matched comparison using the same trained camera-AdaLN pathway would provide stronger evidence. In the DA3 variant of our released system, we have also tested explicit camera conditioning alongside spatial memory, exploring Plücker embeddings, ProPE, and AdaLN. Adding the separate camera pathway increased computational cost while providing only limited improvements in camera control.

Reference

[1] Ting-Bing Xu et al. WorldRoamBench: An Open-World Benchmark for Long-Horizon Stability of Interactive World Models.

Q1: Practical playability requires a component-level latency breakdown, throughput measurements, and peak GPU memory as a function of rollout length.

Response:

Deployment and feedback latency. Objective, independent third-party measurements by the Reactor team show that their optimized deployment of our released DA3-based four-step student achieves approximately 10 FPS on a single B200, with approximately 60 GB peak GPU memory during model loading. By reducing the resolution to 480p or applying NVFP4 quantization, the deployment can achieve approximately 16 FPS for real-time inference. This deployment uses different implementation settings from the H200 diagnostic pipeline. Practical playability remains limited by feedback latency associated with the generation chunk size. We plan to address this through smaller chunks and further deployment optimization.

H200 measurements and teacher–student comparison. We report the available H200 measurements below, covering 16 runs per model. Chunk time is the median of the mean chunk times across runs. HPSv3 is evaluated on eight additional event-conditioned scenes.

MetricFour-step student30-step teacher
Profiled time per chunk (s)6.68328.859
Profiled rollout throughput (FPS)4.7881.109
Full validation-pipeline throughput (FPS)0.7520.495
Peak PyTorch allocated GPU memory (GiB)69.62667.187
HPSv3 ↑0.58730.5961

Component-level latency. The profiled component times for the four-step student are shown below.

ComponentTime per chunk (s)
DiT3.849
In-chunk VAE processing0.129
ViGeo1.904
Spatial rendering and cache updates0.777

Cache growth over long rollouts. Over 60 s of generated content, the student's recorded CPU cache tensor storage grows from 0.212 to 9.115 GiB, with 1.500 GiB of CUDA cache tensors at the end.

Q2: Text-driven events require quantitative evaluation of event success and scene preservation, together with a GenEvent on/off ablation.

Response: We provide a blinded human evaluation and a GenEvent ablation to complement the benchmark aggregates.

Human evaluation of sustained interaction. Our method also ranked first in WorldMark’s Arena human-preference evaluation, providing evidence beyond automatic metrics. We conducted a blinded evaluation of 30 videos of 30 s and 30 videos of 60 s. Each anonymized video was independently assessed by five raters, yielding 300 video-level assessments. Raters were unaware of the model and experimental condition. For each duration, event success is evaluated on 10 videos containing event instructions, scene preservation on revisit is evaluated on 10 videos containing revisits, and trajectory adherence is evaluated on all 30 videos. Failed event cases are retained in the event-success denominator.

Metric30 s60 s
Event success60.0%40.0%
Trajectory adherence63.3%43.3%
Scene preservation on revisit60.0%30.0%

Event success is higher for 30 s videos than for 60 s videos.

GenEvent ablation. To isolate GenEvent’s contribution, we compared models trained with and without GenEvent. Results are reported separately by duration. The 30 s and 60 s test sets each contain 50 samples. Onset delay is measured relative to the instruction boundary and reported only for successful events; failed cases remain in the denominator for event success.

Training settingDurationEvent success ↑Onset delay among successful events (s)Scene preservation ↑
Without GenEvent30 s16.0%0.652.0%
With GenEvent30 s58.0%1.150.0%
Without GenEvent60 s10.0%0.826.0%
With GenEvent60 s36.0%1.830.0%

GenEvent improves event success at both durations. The low scene-preservation scores reflect limitations of the reconstruction modules (DA3 and ViGeo), which struggle to reconstruct scenes after spell-casting events. We plan to explore event-specific fine-tuning of these reconstruction modules in future work.

Reviewer_QfR3OpenReview ↗
Rating
2
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

AlayaWorld is an open, chunk-wise autoregressive video world model built on the video branch of LTX-2.3. Each chunk is conditioned on a persistent appearance anchor, a learned-compressed temporal history (25 RGB frames encoded to 4 latents), a recent-motion latent (the second latent of a 9-frame window), and spatial evidence rendered from a persistent 3D point cache along the requested camera trajectory. Geometry is estimated on generated frames with ViGeo; the cache is scale-aligned to the requested trajectory and rendered at target viewpoints, with invalid regions dropped from the conditioning sequence. Text instructions can be replaced at chunk boundaries to drive events such as spell casting, without resetting visual history. The training mixture combines six real-world/gameplay datasets (222,147 clips total) with GenEvent, a 3,219-clip, 9.46-hour synthetic dataset of instruction-driven events built from Sekai-Real prefixes, Gemini-generated prompts, and Grok Imagine Video synthesis. Training proceeds through bidirectional full-parameter adaptation, history-encoder pretraining plus autoregressive fine-tuning with corrupted histories and prediction-error replay, and four-step autoregressive distillation (distribution matching plus adjacent-noise consistency). The model leads the reported averages on iWorld-Bench (0.7018), WorldMark First-Person Real (82.95), and the WBench consistency aggregate (89.5), with ablations showing the largest drop when spatial memory is removed.

Critical strengths
  1. A clean and well-engineered decomposition of long-horizon context. Separating persistent spatial memory (point-cache rendering at requested viewpoints), compressed temporal history, and a recent-motion latent is a sensible answer to the two different kinds of recall a world model needs: scene identity after a long detour and local dynamics at the chunk boundary. The paper also specifies the unglamorous details that determine whether such a system actually works - causal prefixes, window conventions, invalid-token removal, memory dropout, chunk handoff - and enforces one encoding protocol across the data loader, training, and inference. This train/test consistency is a substantive contribution beyond the architecture diagram.

  2. A reusable data pipeline and an explicit event dataset. The unified annotation schema across six heterogeneous sources is documented with per-source counts and filtering gates, and GenEvent addresses a genuinely sparse category (instruction-driven events) with a defined taxonomy, scene balancing, and manual review. The event categories and duration statistics are a useful resource for later work, and the release of a reproducible pipeline is valuable to the community.

  3. Honest scoping of "playability." The paper states that playability here means navigation plus instruction-driven visual behavior, not a symbolic game state or rule engine, and names the resulting limitation (object state through occlusion). This is the right framing and makes the claims easier to assess.

Critical weaknesses
  1. My own use of this system does not support the reported quality level. I have run this model myself, and the gap between the benchmark leadership reported in the paper and what a user actually sees is large. Interactive translation is the weakest mode, as the paper partly acknowledges, but in my use the view drifts from the requested trajectory well before the reported rollouts end, geometrically inconsistent regions accumulate, and returning to a previously seen area after a detour frequently shows smeared or duplicated structure rather than the same scene. Instruction-driven events fire only some of the time, and when they do the requested object or effect is often approximated rather than realized, while the surrounding scene changes in ways the instruction did not ask for. Long rollouts degrade faster than the consistency numbers imply. Because the headline averages are aggregates over benchmark segments and prompts, they do not capture this user-facing behavior. Since the paper's central claim is interactive long-horizon playability, this is consequential: at minimum, the exact inference configuration used for the reported numbers should be specified, and a human evaluation over 30-60 s interactive sessions (event success, trajectory adherence, scene preservation on revisit) should be reported alongside the benchmark aggregates.

  2. The headline interactive capability - text-driven events - is only shown qualitatively. All three benchmarks measure navigation, trajectory following, or consistency. Spell casting and prompt switching appear only in Figures 4(c), 9, 10, and 11. There is no measurement of whether the requested event occurs, whether it begins at the instruction boundary, whether the requested object/effect (e.g., a fire sphinx vs. a water effect) is the one rendered, or whether the surrounding scene survives the event. Since "playable" is in the title and GenEvent is one of three main contributions, this is the largest gap between claim and evidence, and it is consistent with the unreliable event behavior I observed. A small protocol over a fixed set of initial frames and instructions - event success and prompt adherence judged by humans or a VLM with a reported rubric, temporal alignment to the instruction boundary, and a failure taxonomy - would substantially strengthen the paper. A with/without GenEvent comparison would also substantiate the dataset's contribution, which is currently asserted rather than measured.

  3. The evidence that persistent spatial memory causes the gains is confounded, and system cost is absent. In Table 4, the "w/o spatial memory" variant controls the camera through text prompts instead of trajectory-conditioned renderings, so the reported drops of 17.2 (Interaction) and 16.9 (Consistency) mix removal of cross-chunk recall with removal of explicit geometric camera guidance. The paper mentions a separate camera AdaLN branch that is unused by default, which offers a clean control: keep trajectory conditioning through that branch (or otherwise preserve the camera signal) and remove only the point cache/rendering. As written, the ablation supports "the spatial-conditioning pathway matters," not specifically "persistent spatial memory matters." The head-to-head comparisons are also system-level and differ in backbone and training resources (as Appendix C.2 acknowledges), so the main tables cannot separate the memory design from data and scale. In addition, there are no per-chunk latency, throughput, GPU-memory, or cache-growth numbers and no comparison of the four-step student against the ~30-step teacher; for a model whose selling point is being "playable," this makes deployability impossible to judge. Dynamic content in the cache is likewise unaddressed: moving objects, characters, and the spell effects GenEvent introduces are registered into the persistent geometry and can reappear as ghosts or duplicates when the user returns, which is one of the artifacts I observed in practice.

Questions for authors
  1. What exact inference configuration produced the reported iWorld-Bench, WorldMark, and WBench numbers - the four-step student or the teacher, how many denoising steps, what guidance, and over what rollout lengths? In my own runs I could not approach the reported quality, so a precise, reproducible protocol is needed to interpret the tables.

  2. Were the benchmark scenes and prompts held out from the six-dataset training mixture and from GenEvent? If any evaluation scenes overlap with training data, please report results on a disjoint split.

  3. Can you provide a quantitative evaluation of instruction-driven events? Specifically: event success rate, adherence to the requested effect/object, onset alignment with the chunk boundary, and scene preservation, on a fixed prompt set - plus an ablation with and without GenEvent.

  4. In the "w/o spatial memory" ablation, is the camera still conditioned on the requested trajectory (e.g., via the camera AdaLN branch)? If not, can you add a variant that removes only the point cache while keeping trajectory conditioning, so the effect of persistent memory is isolated?

  5. How are dynamic objects and transient effects handled before registration into the point cache? Are they masked, filtered by motion, or left in? What does a revisit look like after a spell effect was registered?

  6. What is the per-chunk latency and GPU memory of the four-step student at the reported resolution, and how do cache size and latency grow over a 60-second rollout? How much quality is lost relative to the 30-step teacher, and how stable is the median-displacement scale alignment under rotation-only or fast motion over long rollouts?

3 comments

We sincerely thank the reviewer for the detailed and constructive feedback, which has helped us strengthen our experimental evaluation and clarify the capabilities and limitations of our method.

W1&W2&Q3: Interactive reliability and GenEvent’s contribution require stronger quantitative evidence, including human evaluation of 30–60 s sessions, event success and timing, and a controlled dataset ablation.

Response: Thank you for sharing your observations. We provide a blinded human evaluation and a GenEvent ablation to complement the benchmark aggregates.

Human evaluation of sustained interaction. Our method also ranked first in WorldMark’s Arena human-preference evaluation, providing evidence beyond automatic metrics. We conducted a blinded evaluation of 30 videos of 30 s and 30 videos of 60 s. Each anonymized video was independently assessed by five raters, yielding 300 video-level assessments. Raters were unaware of the model and experimental condition. For each duration, event success is evaluated on 10 videos containing event instructions, scene preservation on revisit is evaluated on 10 videos containing revisits, and trajectory adherence is evaluated on all 30 videos. Failed event cases are retained in the event-success denominator.

Metric30 s60 s
Event success60.0%40.0%
Trajectory adherence63.3%43.3%
Scene preservation on revisit60.0%30.0%

Event success is higher for 30 s videos than for 60 s videos.

GenEvent ablation. To isolate GenEvent’s contribution, we compared models trained with and without GenEvent. Results are reported separately by duration. The 30 s and 60 s test sets each contain 50 samples. Onset delay is measured relative to the instruction boundary and reported only for successful events; failed cases remain in the denominator for event success.

Training settingDurationEvent success ↑Onset delay among successful events (s)Scene preservation ↑
Without GenEvent30 s16.0%0.652.0%
With GenEvent30 s58.0%1.150.0%
Without GenEvent60 s10.0%0.826.0%
With GenEvent60 s36.0%1.830.0%

GenEvent improves event success at both durations. The low scene-preservation scores reflect limitations of the reconstruction modules (DA3 and ViGeo), which struggle to reconstruct scenes after spell-casting events. We plan to explore event-specific fine-tuning of these reconstruction modules in future work.

W3&Q4: The original ablation changes both memory access and camera conditioning, leaving the contribution of persistent spatial memory insufficiently isolated.

Response: As stated in the paper, the variant without spatial memory uses text prompts for camera control. Our training includes conditioning dropout to support this text-based camera-control mode. We nevertheless agree that this ablation removes both historical spatial information and trajectory-conditioned rendering, so the reported drops cannot be attributed entirely to persistent memory.

To further examine the role of historical information, we conducted additional experiments on eight scenes, keeping the requested camera trajectories and rendering interface. We either restricted retrieval to the most recent 32 RGB frames or blocked access to pre-reset observations from chunk 27 onward.

ConditionDINO return similarityDA3 geometry scoreEstimated translation alignment
Full reference0.80360.86430.6808
Recent-only retrieval0.69580.75040.6039
Retrieval reset0.69400.78170.4583

Both restrictions reduce performance across all three metrics, supporting the benefit of retrieving older observations.

We agree that a matched comparison using the same trained camera-AdaLN pathway would provide stronger evidence. In the DA3 variant of our released system, we have also tested explicit camera conditioning alongside spatial memory, exploring Plücker embeddings, ProPE, and AdaLN. Adding the separate camera pathway increased computational cost while providing only limited improvements in camera control.

W3&Q6: Assessing deployability requires system-cost measurements, a teacher–student comparison, cache-growth statistics, and an examination of scale alignment under challenging camera motion.

Response:

Deployment and feedback latency. Objective, independent third-party measurements by the Reactor team show that their optimized deployment of our released DA3-based four-step student achieves approximately 10 FPS on a single B200, with approximately 60 GB peak GPU memory during model loading. By reducing the resolution to 480p or applying NVFP4 quantization, the deployment can achieve approximately 16 FPS for real-time inference. This deployment uses different implementation settings from the H200 diagnostic pipeline. Practical playability remains limited by feedback latency associated with the generation chunk size. We plan to address this through smaller chunks and further deployment optimization.

H200 measurements and teacher–student comparison. We report the available H200 measurements below, covering 16 runs per model. Chunk time is the median of the mean chunk times across runs. HPSv3 is evaluated on eight additional event-conditioned scenes.

MetricFour-step student30-step teacher
Profiled time per chunk (s)6.68328.859
Profiled rollout throughput (FPS)4.7881.109
Full validation-pipeline throughput (FPS)0.7520.495
Peak PyTorch allocated GPU memory (GiB)69.62667.187
HPSv3 ↑0.58730.5961

Cache growth over long rollouts. Over 60 s of generated content, the student's recorded CPU cache tensor storage grows from 0.212 to 9.115 GiB, with 1.500 GiB of CUDA cache tensors at the end.

Scale alignment. In our tests, scale alignment remained stable across real-world indoor and outdoor motion sequences, as well as first- and third-person game scenes. When needed, the model’s inference-time camera-motion parameters can be adjusted according to the motion parameters provided for each sample. These adjustments change the camera movement speed and can improve generation quality to some extent. However, the vast majority of tested scenes require no such adjustments.

W1&Q1: Reproducing the reported performance requires clear prompt formats, camera-motion settings, and benchmark-specific inference configurations.

Response:

Prompt format and camera-motion settings in the released interfaces. Using inference prompts consistent with the training format can improve generation quality. For example, “a first-person walk down a misty forest trail” specifies the viewpoint, scene, and intended motion, while camera trajectories are supplied separately as pose data.

For synthetic-trajectory generation in the ViGeo interface, --forward controls translation per frame (default: 0.0049 in trajectory-coordinate units), while --yaw controls yaw rotation per frame (default: 0.0 degrees). In the DA3 interactive interface, motion.forward_units_per_second and motion.yaw_degrees_per_second specify the corresponding speed scales, with defaults of 1.905 coordinate units/s and 9.375 degrees/s, respectively. Separately, DA3's control.action_scale rescales the translational and rotational action signals supplied to the model without modifying the input trajectory; its default is 0.1337,0.2769,2.021,0.00442,0.01026,0.00448.

Small, sample-specific adjustments to camera speed and action scaling can improve camera-control performance and generation quality.

Benchmark-specific inference settings. The settings for the four-step student are summarized below.

ConfigurationWBenchWorldMarkiWorld-Bench
Denoising steps444
CFG scale1.01.01.0
STG guidanceDisabledDisabledDisabled
Random seed424242
Video duration (s)7.71–23.7119.71 / 39.71 / 59.713.375
Video frames185 / 281 / 377 / 473 / 569473 / 953 / 143381

Prompt conversion and benchmark-specific preparation. We use a fixed rewriting prompt to convert the benchmark prompts into the format used during training.

For WBench, prompt preparation also incorporates per-turn action descriptions and physics clauses derived from the benchmark’s causal_fidelity.dims annotations. We use the subject masks provided by the benchmark to anchor the first-frame foreground in the warped spatial-conditioning images. The masks are dilated by 12 pixels to exclude the foreground from the static geometry cache.

For WorldMark, we follow the same target evaluation durations of 20, 40, and 60 seconds used for the other models. Our four-step student produces 473, 953, and 1433 frames, respectively, for one-, two-, and three-command sequences.

For iWorld-Bench, we use a 48-frame static-pose prefix for initialization, take the trajectory window [48:129], and retain the first 81 generated frames for evaluation.

Q2: Are the benchmark scenes and prompts held out from the six-dataset training mixture and GenEvent?

Response: We strictly exclude all benchmark evaluation samples from the data used for our training and fine-tuning, including both the six-dataset mixture and GenEvent.

At evaluation time, we use GPT with a single fixed rewriting instruction to convert the benchmark prompts into the prompt format used during training. This follows established prompt-adaptation practices in video generation, as also used in systems such as CogVideoX. The conversion is performed only during evaluation and does not involve training our model on the test data.

The evaluation prompts may therefore resemble our training captions in format, but neither the test samples nor their rewritten prompts were used for training or fine-tuning.

W3&Q5: Dynamic objects and spell effects may enter the persistent spatial cache and cause ghosting or duplicated structures on revisits.

Response: Masking dynamic objects and reconstructing only the static background can mitigate this issue. For WBench, we use the benchmark-provided subject masks, dilated by 12 pixels, to exclude foreground regions from the static geometry cache. This prevents the masked foreground regions from being stored as persistent scene geometry and helps reduce ghosting and duplicated structures on revisits.

Short-lived spell effects, such as flashes or smoke, may appear outside the provided subject masks. DA3 and ViGeo also struggle to reconstruct scenes after spell-casting events, which contributes to the low scene-preservation scores.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.