acceptodds
Under review as a conference paper at ICLR 2027

WorldPlay2: Extending Real-Time Interactive World Models in Control and Horizon

11 views

Abstract

Interactive world models require responding in real time to versatile controls and maintaining long-horizon consistency. However, modeling heterogeneous controls remains difficult, while explosive contexts and unstable distillation impede achieving both long-horizon consistency and real-time responsiveness. In this paper, we present WorldPlay2, an interactive world model that couples a factorized hybrid control interface with a co-design of compressed memory and stable distillation. 1) Our factorized hybrid control interface integrates frame-aligned action control with structured semantic control that explicitly disentangles scene appearance, character identity, and dynamic semantic events, thereby facilitating effective control learning. 2) To achieve efficient long-horizon modeling, we compress historical contexts into compact memory tokens shared by the autoregressive student and the bidirectional teacher. This design enables clip-wise, memory-conditioned score evaluation instead of jointly processing an entire long rollout, substantially reducing distillation overhead. 3) We further propose Stable Forcing, which initializes the autoregressive student via a few-step strategy and leverages full-rollout replay to preserve the quality of long-horizon rollouts, ensuring robust and stable distillation. Extensive experiments demonstrate the strong generalizability of our model and its superior performance compared to existing methods.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
World Models, Diffusion Models

Reviews

Among:
Shape:
1234this paper, Top 17%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.75 over 4 reviews, above 83% of the others.
Reviewer_MUJaOpenReview ↗
Rating
4
Confidence
4
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

The paper introduces WorldPlay2, an interactive video world model that supports both navigation and semantic events while maintaining consistency over longer rollouts. It separates low-level actions from text descriptions of the scene, character, and event to make the controls easier to learn. Historical video is compressed into memory tokens used by both the student and teacher, allowing the teacher to evaluate shorter clips without processing the full rollout at once. The authors also introduce Stable Forcing, which uses PDD for few-step initialization, then runs complete student rollouts and replays selected denoising steps for gradient updates. The main contribution is how the memory and distillation procedures work together to make long-rollout training more efficient and stable. Results on WBench, RevisitBench, and interaction tests show improvements in control and consistency, with streaming generation reaching 16 FPS on eight H20 GPUs.

Critical strengths
  1. The combination of compressed memory and Stable Forcing is a strong part of the paper. The memory design makes long histories cheaper to use during both generation and teacher evaluation, while the distillation procedure helps preserve quality over longer rollouts. The revisit results and qualitative examples provide encouraging evidence that this combination improves long-term consistency. This addresses a practical difficulty in current world models and is my main reason for viewing the work positively.

  2. The engineering work also looks solid. The paper brings together a multi-stage training pipeline, caching, quantization, and a lightweight VAE, and provides a latency breakdown for the resulting streaming system. The reported 16 FPS on eight H20 GPUs supports its practical value. The examples shown in the paper look good, and the method and training workflow are explained clearly, making the overall contribution easy to follow.

Critical weaknesses
  1. Could the authors provide the model, training, and evaluation settings, including the base model and its size, output resolution, training parameters for each stage, and inference sampling settings? Are there plans to release the training and evaluation pipelines and datasets? If some parts cannot be shared, please clarify what will be available for reproducing the results.

  2. Which VLM version was used for the interaction evaluation, and what were the evaluation settings, including video sampling, model parameters, and score aggregation? Have the authors evaluated the same samples with other VLMs to check whether the reported gains and model rankings remain consistent? This would help show how sensitive the results are to the choice of evaluator.

Questions for authors

Please refer to the two questions in the Weaknesses section.

3 comments
AuthorsOpenReview ↗

Rebuttal for Reviewer MUJa (Part 1)

We sincerely thank the reviewer for the positive feedback on our work and the valuable comments. Below, we address your concerns point by point:

W1: Could the authors provide the model, training, and evaluation settings......

Our base model is Wan2.2-A14B with a resolution of 448×832, and the detailed training configurations and inference settings are provided in Table 5 of the revised Appendix. Regarding open-sourcing, while the training dataset cannot be publicly released due to copyright restrictions, we will release the inference code, along with the model weights from each training stage, to support the reproduction of our results.

W2: Which VLM version was used for the interaction evaluation......

We use Doubao-Seed-2.1-Pro-260628 as our VLM evaluator, whose model parameters are not publicly disclosed as it is a proprietary model. For each video, we sample frames at 3 FPS, and the final score is computed by aggregating the scores across the two dimensions described in the Appendix, averaging them over all test samples, and finally normalizing the result to a 0–100 scale. To verify the sensitivity to different VLM evaluators, we also conduct an evaluation using Gemini-3.8-Flash under the same protocol. Although there are slight variations in the model rankings, the average scores still clearly demonstrate the superiority of our method.

Seed-2.1-ProGemini-3.8-Flash
MethodAvg. ↑EO. ↑CI. ↑Avg. ↑EO. ↑CI. ↑
WorldPlay43.347.938.745.448.542.3
AlayaWorld39.448.330.539.947.832.0
Lingbot-World-V252.561.543.455.363.746.9
EchoWM43.856.331.244.154.733.5
Alaya-Evoke-Turbo40.745.735.739.542.836.1
Ours (full)74.779.270.174.977.472.3

Thank you for the detailed response and additional experiments. The clarified experimental settings and the evaluation using a second VLM have addressed my main concerns. Overall, I consider this a solid piece of work and appreciate the authors’ efforts to strengthen the paper.

We sincerely thank you for the review and the suggestions.

Best regards,

Authors

Reviewer_Lp24OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes an interactive world model called WorldPlay2, which includes designs of compressed memory and stable distillation. Authors mention three points for their contribution: factorized hybrid control interface, compressed context for efficient long-horizon modeling, and Stable Forcing. Experiments show that the proposed method has strong generalizability and superior performance over previous methods.

Critical strengths
  • The proposed system design addresses practical bottlenecks in video world models. The method addresses the coupled challenges of control, long-term memory, and efficient generation.

  • The training dataset is large and the qualitative results are impressive. The overall performance of WorldPlay2 seems good.

  • Performance on revisit consistency and event responsiveness is strong. The proposed system improves on RevisitBench PSNR and Met3r, etc.. The improvements support the paper's objecties of maintaining world consistency and responding to interactive events

Critical weaknesses
  • The abstract is insufficiently self-contained and obscures the paper’s motivation. It introduces “unstable distillation” as a central bottleneck before explaining why distillation is needed for real-time interactive generation. It then invokes an “autoregressive student,” a “bidirectional teacher,” “score evaluation,” and “full-rollout replay” without establishing their roles or providing sufficient intuition. Consequently, readers must already understand the underlying training framework to follow the claimed contributions.

  • The novelty of this work is insufficiently addressed. The history compressor follows TinyHistory, initialization uses PDD, gradient replay follows RELIC, and teacher–student memory alignment is from WorldPlay. The contribution therefore appears primarily to be the adaptation and integration.

  • Lack of ablation study. Table 8: the configuration retaining initialization while removing full-rollout is missing. These results demonstrate the importance of initialization within the tested configuration, but do not establish the independent benefit of full-rollout after initialization.

  • Also, the justification for the gradient approximation in full-rollout replay is insufficient. Algorithm 1 computes score differences on fully denoised outputs but backpropagates through clean predictions from randomly selected intermediate steps. This generally yields a surrogate gradient rather than the gradient through the final generated output. Comparisons against final-step-only backpropagation and, on shorter rollouts, full backpropagation would help justify the design.

  • The contribution of the factorized control interface to the final system’s event responsiveness is insufficiently isolated. Adding event data substantially improves the teacher’s event response score (table 5), whereas the interface ablation only evaluates teacher navigation accuracy (Table 6).

Questions for authors
  • Will the datasets and weights be fully open-sourced? If not, how hard is it to reproduce the results in the paper?

  • What GPUs and how many GPU hours are used for the training?

3 comments
AuthorsOpenReview ↗

Rebuttal for Reviewer Lp24 (Part 1)

We sincerely thank the reviewer for the time and the valuable comments. Below, we address your concerns point by point:

W1: The abstract is insufficiently self-contained and obscures the paper’s motivation.

We thank the reviewer for pointing this out. In the abstract, we have added a clear explanation of why distillation is needed for real-time interactive generation. Specifically, the autoregressive student refers to our final interactive world model, while the bidirectional teacher denotes the model used to guide the student. Score evaluation is used to measure the discrepancy between the generation distributions of the student and the teacher, and full-rollout replay refers to the rollout and update strategy for the student model. We have revised the abstract accordingly in the revised paper.

W2: The novelty of this work is insufficiently addressed.

Our technical novelty lies in two main aspects, targeting two fundamental bottlenecks in real-time interactive world models. First, regarding interactivity, prior works struggle to unify low-level action control and high-level semantic control. We propose a factorized hybrid control interface to effectively disentangle these distinct controls, ensuring precise responsiveness across different levels. Second, regarding long-horizon consistency and real-time efficiency, we propose a co-design of memory and distillation. Our compressed memory is not only designed for long-context modeling, but crucially makes long-horizon distillation computationally feasible while maintaining consistency. Moreover, to overcome the training instability and performance degradation we observed in long-horizon distillation, we propose few-step initialization with full-rollout replay. Thus, although inspired by prior arts, our framework provides a systematic solution for interactive world models rather than a naive combination.

W3: Lack of ablation study.

We add an ablation study on initialization without full-rollout replay, with the results presented in the table below (and updated in Table 10 of the revised paper). The results demonstrate that with few-step initialization alone, the model can already undergo stable distillation training and achieve competitive performance. Incorporating full-rollout replay yields a noticeable improvement, confirming its critical role in long-horizon distillation.

MethodAvg. ↑Qua. ↑Set. ↑Int. ↑Con. ↑Phy. ↑
w/o init & full-rollout73.974.464.681.982.665.8
w/o init75.276.268.082.683.365.9
w/o full-rollout79.879.577.687.386.667.9
Final-step BP79.480.276.186.887.766.1
Full-context Teacher82.381.679.588.189.472.9
Ours (full)83.181.881.588.390.074.0

W4: Also, the justification for the gradient approximation in full-rollout replay is insufficient.

We thank the reviewer for this insightful observation. Indeed, we utilize a surrogate gradient to update the model. To demonstrate the effectiveness of our approach, we compared it against the final-step-only backpropagation baseline. As shown in table of our response to W3, our surrogate gradient effectively backpropagates signals across different timesteps, achieving superior performance. Furthermore, we also attempted full backpropagation on shorter rollouts. However, the excessively long computation graph results in Out-of-Memory errors.

W5: The contribution of the factorized control interface to the final system’s event responsiveness is insufficiently isolated.

We thank the reviewer for pointing this out. To further demonstrate the effectiveness of our factorized hybrid control, we have conducted additional evaluations for Table 5 and Table 6 of the original paper, with the updated results presented below. The results demonstrate incorporating structured semantic control significantly enhances the model's responsiveness to high-level semantic events. Furthermore, incorporating a small fraction of interactive data does not noticeably compromise navigation performance.

MethodAvg. ↑EO ↑CI ↑Rot. Err. ↓Trans. Acc. ↑
w/o interactive event dataset46.757.535.80.15797.9
w/o structured semantic control70.271.468.90.47888.9
Ours (Bidirectional)84.086.881.20.16898.3
AuthorsOpenReview ↗

Rebuttal for Reviewer Lp24 (Part 2)

Q1: Will the datasets and weights be fully open-sourced? If not, how hard is it to reproduce the results in the paper?

We plan to open-source our model weights to support the reproduction of our results. Although the dataset cannot be publicly released due to copyright restrictions, we have detailed the training pipeline and configurations (Table 5) in the Appendix, and we will release the model checkpoints across different training stages to facilitate reproducibility.

Q2: What GPUs and how many GPU hours are used for the training?

We trained our model using NVIDIA H20 GPUs, consuming approximately 134.4K GPU hours in total.

Dear Reviewer Lp24,

As the discussion period closes in a few hours, we would greatly appreciate it if you could let us know whether our response has addressed your concerns. We are happy to clarify anything further.

Thank you for your time.

Best regards, Authors

Reviewer_FxmkOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper presents WorldPlay2, an interactive video world model that combines navigation control, semantic interactions, and long-horizon generation. The control interface separates frame-aligned actions from textual descriptions of the scene, character, and event. A compressed history representation is shared by the autoregressive student and bidirectional teacher, allowing the teacher to evaluate long student rollouts in shorter clips. The proposed Stable Forcing procedure combines PDD initialization with complete few-step sampling during rollout and gradient replay at randomly selected intermediate denoising steps. Experiments cover the navigation split of WBench, a revisit benchmark, and semantic interaction tasks. The paper reports improvements over the evaluated baselines and a generation throughput of 16 FPS on eight H20 GPUs.

Critical strengths
  1. Using compressed history for both generation and teacher evaluation addresses a practical training bottleneck. The memory representation allows the student to train on long rollouts without requiring the teacher to process the entire sequence at full resolution. The efficiency results in Section 4.2 and Figure 6 support the usefulness of this design. Table 7 also provides a useful comparison at the same training length: compression incurs some loss in SSIM and LPIPS while retaining competitive PSNR and MEt3R. This is a reasonable trade-off for extending the training horizon.

  2. The improvements in semantic interaction and revisit consistency are substantial. Table 2 reports an interaction score of 74.7, compared with 52.5 for the strongest listed baseline. On RevisitBench, the method improves all four reported metrics; for example, PSNR increases from the best baseline’s 17.05 to 19.71, and MEt3R decreases from 0.179 to 0.105. These results provide meaningful evidence for the value of the complete system. The quantitative ablations in the appendix also help assess the effects of the training data and individual design choices.

Critical weaknesses
  1. The experiments do not yet isolate the main methodological gains. The paper builds on TinyHistory for compression, PDD for initialization, and a replay strategy related to RELIC. The contribution therefore depends substantially on what the proposed adaptations add to the combined system. For semantic control, Table 5 shows a large improvement from adding interaction data, from 46.7 to 84.0 on the bidirectional model. However, the structured-control ablation in Table 6 only measures navigation. It does not establish whether structuring the captions improves semantic interaction when the training data and textual information are held fixed.

    A similar issue arises in Table 8. The comparison includes neither component, replay without initialization, and the full method, but omits initialization without replay. This leaves the benefit of replay after a good initialization unresolved. Moreover, the final scores and examples in Figure 6(c) demonstrate degraded output quality, but provide limited evidence for the more specific claims about training stability and mode collapse. These omissions matter because structured control and Stable Forcing are presented as central contributions.

  2. Important details of the distillation update and experimental configuration remain unclear. In Section 3.3, the score difference is evaluated on the output of the complete four-step sampler, whereas gradients are replayed through a prediction at a randomly selected intermediate step. I could not determine the precise surrogate objective from Eq. (7) and Algorithm 1. The paper should explain how the final-sample score difference is applied to the intermediate prediction and which dependencies are detached. The relationship between the velocity parameterization in Eq. (5) and the score outputs in Eq. (7) also needs to be specified. In Algorithm 1, line 23 should explicitly recover the appropriate intermediate noisy state from the snapshot.

    The backbone, resolution, and principal training settings are also not stated clearly enough to reproduce the results. For deployment, Table 4 provides useful throughput measurements, but it is unclear whether Tables 1–2 use the same quantized model and lightweight VAE as the final speed measurement. The reported 0.998 seconds per chunk does not establish the delay between a new control input and its visible effect. These details are needed to assess the claimed combination of quality and real-time interaction.

The system-level results are promising, but these concerns leave me slightly below the acceptance threshold. A precise description of the update, together with the missing controlled comparisons, could change my assessment.

Questions for authors
  1. Could you report semantic interaction scores for structured and unstructured captions using the same training data, textual information, and training budget? For Table 8, what is the result when PDD initialization is retained but full-rollout replay is removed? These comparisons would help establish the contribution of the proposed control and distillation changes.

  2. What exact loss is differentiated during gradient replay? Please specify how the score difference evaluated on the final sample is used at the cached intermediate step, whether the outputs in Eq. (7) have already been converted from velocities to scores, and which noisy state is used in line 23 of Algorithm 1.

  3. Were the quality results in Tables 1–2 obtained with the final deployment configuration in Table 4, including quantization and LightVAE? Please also provide the backbone, output resolution, and the measured delay from a control change to its first visible effect.

3 comments
AuthorsOpenReview ↗

Rebuttal for Reviewer Fxmk (Part 1)

We sincerely thank the reviewer for the time and the valuable comments. Below, we address your concerns point by point:

W1: The experiments do not yet isolate the main methodological gains.

We thank the reviewer for pointing this out. To further demonstrate the effectiveness of our factorized hybrid control, we have conducted additional evaluations for Table 5 and Table 6, with the updated results presented below. The results demonstrate incorporating structured semantic control significantly enhances the model's responsiveness to high-level semantic events. Furthermore, incorporating a small fraction of interactive data does not noticeably compromise navigation performance.

MethodAvg. ↑EO ↑CI ↑Rot. Err. ↓Trans. Acc. ↑
w/o interactive event dataset46.757.535.80.15797.9
w/o structured semantic control70.271.468.90.47888.9
Ours (Bidirectional)84.086.881.20.16898.3

We added an ablation study on initialization without full-rollout replay, presented in the table below and updated in the revised paper. The results demonstrate that with few-step initialization alone, the model can already undergo stable distillation training and achieve competitive performance. Incorporating full-rollout replay yields a noticeable improvement, confirming its critical role in long-horizon distillation.

MethodAvg. ↑Qua. ↑Set. ↑Int. ↑Con. ↑Phy. ↑
w/o init & full-rollout73.974.464.681.982.665.8
w/o init75.276.268.082.683.365.9
w/o full-rollout79.879.577.687.386.667.9
Full-context Teacher82.381.679.588.189.472.9
Ours (full)83.181.881.588.390.074.0

By “mode collapse,” we originally intended to describe a failure mode in which noise-like artifacts appear on the ground. To avoid confusion, we have replaced “mode collapse” with “degrading generation quality,” which more precisely describes our observation and maintains consistency with the Introduction. To further verify training stability, we provide qualitative comparisons between “w/o init & full-rollout” and our method at the early, middle, and late stages of distillation (Fig.7 in the revised paper). We observe that the “w/o init & full-rollout” baseline exhibits severe visual degradation at the early stage of training, and fails to recover from this issue during the middle and late stages. In contrast, our method continuously improves generation quality as training iterations progress, demonstrating its better training stability.

W2: Important details of the distillation update and experimental configuration remain unclear.

We apologize for the confusion. Our training objective is derived from Self Forcing, optimizing the reverse KL divergence,

∇θDKL=Eϵ∼N(0,I),σ∼[0,1][−(sreal(xσ,σ)−sfake(xσ,σ))dNdθ],x=Nθ(ϵ).\nabla_{\theta} D_{\mathrm{KL}}=E_{\epsilon \sim \mathcal{N}(0,I), \sigma \sim [0,1]} \left[-\left(s_{\mathrm{real}}(x^{\sigma}, \sigma)-s_{\mathrm{fake}}(x^{\sigma}, \sigma)\right)\frac{dN}{d\theta}\right], \quad x=N_{\theta}(\epsilon).

Here, we omit other inputs for brevity. xσx^{\sigma} denotes the noised rollout samples, while sreals_{\mathrm{real}} and sfakes_{\mathrm{fake}} represent the scores computed by the real and fake score models, respectively. The score computations are performed without gradient tracking (i.e., detached, corresponding to Stage 2 in Algorithm 1). For our full-rollout replay, to reduce GPU memory consumption, the rollout process itself is also detached (Stage 1 in Algorithm 1). However, once the scores are computed, we re-execute a gradient-enabled forward pass to backpropagate the gradients and update the student model (Stage 3 in Algorithm 1).

For flow matching models, the relationship between the score and the velocity is given by:

s(xσ,σ)=−xσ+(1−σ)v(xσ,σ)σ.s(x^{\sigma},\sigma)=-\frac{x^{\sigma}+(1-\sigma)v(x^{\sigma},\sigma)}{\sigma}.

Consequently, the score difference can be formulated as:

sreal−sfake=−1−σσ(vreal−vfake).s_{\mathrm{real}} - s_{\mathrm{fake}} = -\frac{1-\sigma}{\sigma}(v_{\mathrm{real}}-v_{\mathrm{fake}}).

This implies that the score difference is equivalent to the velocity difference multiplied by a scaling factor. In practice, during training, we follow the implementation from Self-Forcing (https://github.com/guandeh17/Self-Forcing/blob/33593df3e81fa3ec10239271dd2c100facac6de1/model/dmd.py#L54 ). We have clarified and revised the training objectives, Eq. 7, and Algorithm 1 in the paper accordingly.

Our base model is Wan2.2-A14B with a resolution of 448×832, and the detailed training configurations are provided in Table 5 of the revised paper. For Table 1 and Table 2, we evaluate our method using the quantized model without LightVAE. Additionally, the latency from receiving a new control input to decoding the first corresponding frame is 0.982s (partial reduction in VAE decoding time).

AuthorsOpenReview ↗

Rebuttal for Reviewer Fxmk (Part 2)

Q1: Could you report semantic interaction scores......

We conduct the training and evaluation under the exact same settings, and the results are presented in our response to W1. Additionally, the results for initialization without full-rollout replay are also shown in our response to W1.

Q2: What exact loss is differentiated during gradient replay?

Please refer to our response to W2, where we have provided a detailed explanation regarding the training objective, the computation and usage of the scores, the relationship between score and velocity. The noisy state is used in line 23 of Algorithm 1 is recovered from the Snapshot in line 8 of Algorithm 1. We have also revised the paper accordingly.

Q3: Were the quality results in Tables 1–2 obtained with the final deployment configuration in Table 4.

Please refer to our response to W2.

Dear Reviewer Fxmk,

As the discussion period closes in a few hours, we would greatly appreciate it if you could let us know whether our response has addressed your concerns. We are happy to clarify anything further.

Thank you for your time.

Best regards, Authors

Reviewer_rPNEOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper presents WorldPlay2, a video world model supporting real-time interaction and long-term consistency. Methodologically, the authors decouple frame-level action control from semantic control over scenes, characters and events to enable more flexible interaction. Meanwhile, they introduce compressed memory to reduce the cost of long-sequence generation and distillation. They also propose Stable Forcing, which leverages few-step initialization and full-rollout replay to improve the stability of long-horizon generation. The experiments are reasonably comprehensive, achieving promising results on WBench, RevisitBench and complex interactive event benchmarks. Ablation studies also verify the effectiveness of the core designs.

Critical strengths
  1. The decomposition of control signals is reasonably designed. The paper separates frame-level actions from high-level semantic control, and further divide semantic control into scene, character and event. I find this design intuitive. It mitigates interference among different control signals and enables the model to better support complex interactions across diverse characters and scenes. The empirical improvements are also substantial.

  2. The joint design of compressed memory and distillation is valuable. This is one of the contributions I appreciate most. The authors not only address the computational cost of long-sequence generation, but also make compressed memory directly applicable for chunked evaluation by the teacher model. It retains long-range historical context while cutting distillation overhead. This joint design is practical and exhibits good transferability.

  3. The Stable Forcing design is convincing. The authors first obtain a stable few-step student model via PDD, then use Full-rollout Replay to decouple the full generation trajectory from gradient computation. This approach preserves the quality of long-horizon generation as much as possible while reducing training costs. The corresponding ablation results clearly demonstrate the efficacy of these components, serving as key evidence to validate the proposed method.

Critical weaknesses
  1. The reliability of compressed memory over even longer time horizons remains unclear. While RevisitBench demonstrates favorable geometric consistency, it remains an open concern whether the model can reliably remember previously appearing objects, appearance details and local spatial relationships after drastic compression of historical information. Current evaluations mostly reflect global consistency, and there is a lack of direct evidence to verify the reliability of such fine-grained memory.

  2. The reliability on complex interactions and real-world scenarios should be treated with caution. Complex events are mainly assessed automatically via VLM. Although such evaluations can judge whether actions follow the given instructions, they may not adequately capture physical plausibility such as contact, collision and object dynamics. The scores themselves may also suffer from inherent biases. Besides, a large portion of the training data is sourced from games and simulated environments, leaving uncertainty regarding the model’s generalization capability on more complex real-world scenes.

Questions for authors

The questions below are mainly intended to discuss design choices and empirical behaviors with the authors, and do not require additional experiments.

  1. On the long-term reliability of compressed memory. I am curious about to what extent the model can preserve consistency in appearance, identity and spatial position for objects that appear long ago and then re-enter the view, after the historical information is compressed by around 32×. Besides, while the authors have structurally decomposed the input semantics, the historical memory still relies primarily on implicit feature compression instead of explicitly recording object or scene structures. I wonder why this design is adopted. Have the authors considered incorporating structured memory? Could you discuss the pros and cons of these two paradigms?

  2. On physical consistency for complex events. I am interested in whether Stable Forcing leads to noticeable changes in the model’s adherence to physical laws. For complex events such as grasping, collision and object motion, is the final few-step autoregressive model more prone to physically implausible artifacts compared with the teacher model before distillation? If the model is deployed for real-world interaction, what directions do the authors think can further improve the reliability of such events?

  3. On the fusion and conflict of different control signals. In Frame-aligned Action Control, the encoding of continuous rotation signals is directly added to the embeddings of discrete actions. I am curious why addition is chosen instead of other fusion strategies such as concatenation or gating. Could this fusion cause mutual interference when encountering rare action combinations? Furthermore, what happens if low-level action control conflicts with high-level semantic events? For instance, when the semantic instruction asks the character to stop and enter the car while the navigation control keeps commanding forward movement. How does the model resolve such conflicts? Is there a priority mechanism among different control signals?

  4. I appreciate the substantial efforts the authors have devoted to the structured design of control signals. However, I am curious that the current ablation only removes Structured Semantic Control. If the same scene, character and event information are retained but merged into a plain caption instead of being explicitly split into three fields, how large would the performance gap be? I would like to understand to what extent the improvements stem genuinely from this factorized design, rather than merely from richer semantic supervision.

4 comments
AuthorsOpenReview ↗

Rebuttal for Reviewer rPNE (Part 1)

We sincerely thank the reviewer for the positive feedback on our work and the valuable comments. Below, we address your concerns point by point:

W1: The reliability of compressed memory over even longer time horizons remains unclear.

From the perspective of our training mechanism, the history compressor undergoes a first-stage pre-training designed to reconstruct historical information directly from the compressed memory. This process effectively forces the model to learn how to retrieve relevant information from the compressed representations, thereby significantly enhancing the reliability of its fine-grained memory (as demonstrated in Figure 4 of TinyHistory). To further verify whether compressed memory can retain consistency regarding objects, appearance, and local spatial relationships, we followed the evaluation protocol of TinyHistory and utilized a VLM (doubao-seed-2-1-pro-260628) to assess the three dimensions on WBench. The results are presented in the table below, demonstrating that our compressed memory preserves these fine-grained details better than previous methods.

ModelAverageObjectAppearanceLocal Spatial
WorldPlay70.969.874.568.4
AlayaWorld53.951.560.749.5
Lingbot-World-V258.657.563.954.3
EchoWM57.152.965.852.5
Alaya-Evoke-Turbo71.876.172.367.1
Ours85.887.988.181.4

W2: The reliability on complex interactions and real-world scenarios should be treated with caution.

To more reliably evaluate physical plausibility in complex interactions and real-world scenarios, we conduct a human evaluation based on user preference (ELO rating, base 1200, K=32) on the interactive test cases, asking human evaluators to assess preferences across contact, collision, and object dynamics. We incorporate these results into Table 3 of the revised paper, which further demonstrates that our method achieves superior physical plausibility.

MethodPP. (ELO) ↑
WorldPlay1088
AlayaWorld1141
Lingbot-World-V21256
EchoWM1277
Alaya-Evoke-Turbo1030
Ours1408

As for real-world generalization, because our framework leverages a pretrained video diffusion model, our empirical results ( Fig. 1 (1st case), Fig. 5 (first two cases), Fig. 6 (top-left case), and Fig. 11 (first four cases) in the revised paper) show that our model successfully retains the base model's generalization ability in real-world scenes. Meanwhile, we agree that increasing the proportion of real-world training data could further boost generalization. However, curating and annotating high-quality real-world data tailored for interactive world models remains a non-trivial challenge, which we now discuss in the Limitations.

Q1: On the long-term reliability of compressed memory.

We observe that our model effectively maintains consistency in appearance, identity, and spatial position for durations of around 1 minute. However, performance degrades over longer horizons (e.g., 1 min 30 s). In particular, identity in third-person scenarios may exhibit inconsistencies. In our observation, the compressed memory is both efficient and scalable, and the model's performance can be further improved by collecting longer-horizon data with more complex trajectories.

For structured memory, current methods typically rely on explicit 3D representations, which provide strong conditioning during rollouts to preserve fine-grained details. However, in long-horizon, large-scale rollouts, incrementally updating the 3D representation often causes misalignment, making it difficult to maintain consistent 3D representation and ultimately resulting in performance degradation. Furthermore, this paradigm struggles to handle dynamic content effectively. Conversely, implicit compressed memory serves as a highly scalable paradigm. It implicitly learns long-horizon geometric consistency from data without suffering from intractable memory updates, though at the cost of requiring substantially more training data and compute.

Q2: On physical consistency for complex events.

As for Stable Forcing, it optimizes the reverse KL divergence and thus exhibits mode-seeking behavior. Consequently, for complex events where the teacher model already performs reasonably well, distillation typically yields a noticeable improvement. However, for cases where the teacher model severely struggles (i.e., rarely succeeds), distillation can hardly improve performance. Therefore, for real-world deployment, we believe that improving the teacher model's capability on such complex events is a more effective approach.

AuthorsOpenReview ↗

Rebuttal for Reviewer rPNE (Part 2)

Q3: On the fusion and conflict of different control signals.

In our initial experiments, we tried various architectural designs and found that simply decoupling the encoding of continuous and discrete actions is sufficient to achieve strong performance. This is because continuous and discrete actions have distinct numerical scales, and thus employing separate encoding facilitates model learning. As for how the encoded features are subsequently fused, we only observed sightly differences. Therefore, we opted for simple addition. Furthermore, as shown by the visualizations in the supplementary videos, our model responds well to both rare and complex action combinations, and we have not observed noticeable mutual interference.

In cases where low-level actions conflict with high-level semantic events, our model resolves the ambiguity by prioritizing controls based on the current state. For instance, when the character is distant from a vehicle, the model executes the low-level actions, whereas it triggers the high-level semantic event when in close. This adaptive behavior is an advantage of our data-driven training, enabling the model to automatically execute the more feasible option.

Q4: However, I am curious that the current ablation only removes Structured Semantic Control.

For the w/o structured semantic control ablation, we utilize a single VLM-annotated plain caption that combines the scene, character, and event descriptions. Our experiments show that this merging makes it harder for the model to disentangle individual concepts, resulting in weaker responsiveness to interactive controls (see the table below and Table 7 in the revised paper).

MethodAvg.EO.CL.
w/o interactive event dataset46.757.535.8
w/o structured semantic control70.271.468.9
Ours (Bidirectional)84.086.881.2

Thank you for the supplementary detailed experiments and clarifications which address some of my concerns, so I will retain my original score.

Thanks for your positive feedback! We are glad that our response has addressed your concerns.

Comments on the paper

2 comments

Dear Reviewers, 

We’re about halfway through the discussion period. Please read the authors’ rebuttal and assess whether it adequately addresses the key concerns raised in your review. We encourage you to leave a follow-up comment for the authors, indicating which concerns have been resolved and whether any important issues remain or require further clarification. Please update your review and scores as appropriate in light of the rebuttal and discussion.

Thank you for your time and participation. 

Best regards, 

AC

Dear Reviewers,

Thank you again for your review. We have provided our responses and additional results about your concerns, and would appreciate your thoughts when you have a chance to review them. Please let us know if any points remain unclear or require further clarification.

Thanks for your time and consideration.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.