acceptodds
Under review as a conference paper at ICLR 2027

Generalizable World Simulator: Real-Time & Long-Horizon Interaction with Diverse Objects

Abstract

Interactive world models have shown great potential in serving as a simulator for robot manipulation. However, existing models are either overfit to a single environment or lack ability to simulate long-horizon contact-rich events. We present Generalizable World Simulator (GWS), a world model whose dynamics model is trained from scratch on a large robot-native teleoperation dataset, with no internet-video pretraining. By focusing on developing the optimal model architecture and learning recipe to fit robot-native data, we managed to learn a world model that is orders of magnitude smaller than SOTA video generation models, yet shows significantly better understanding of contact and causality, and runs 10 FPS on an commercial GPU. Experimentally, our model significantly outperforms previous SOTA methods across various tasks including future prediction, policy evaluation, and mixed-data co-training. We further demonstrated various potential applications where our model can be used in a robotics setting.

Primary area:
applications to robotics, autonomy, planning
Keywords:
world model, action-conditioned video generation, policy evaluation, real-time rendering

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 2 reviews, above 27% of the others.
Reviewer_Usd5OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This work presents an action-conditioned generative model called GWS. It addresses the error accumulation problem encountered by previous methods during long-horizon generation. The work makes three key contributions: (1) a memory mechanism combining short-term and long-term memory, (2) a keyframe selection mechanism for long-term memory, and (3) a model architecture. The results are promising and establish a SOTA.

Critical strengths

The results look good, especially the live demo. However, these examples use fixed inputs, so it is unclear how well the model generalizes to other OOD cases.

Critical weaknesses
  1. The manuscript appears to be at an early stage and contains numerous errors, such as “ABD datasets()” on L261, and “SSDD-M()” and “decoder ()” in the implementation details.
  2. The rationale behind all design choices is missing. For example, why use unit-sphere normalization, VLM-based supervision for contact prediction, Diffusion Forcing, a learned sink frame?
Questions for authors
  1. Why does error accumulation occur when an action serves as a condition? Does it refer to an appearance shift or a geometric shift? An illustrative figure would help clarify this.

  2. The rationale behind the unit-sphere tokenizer is unclear. Why use unit-sphere normalization? Is this normalization applied to ztz_t? I am unsure because you refer to zz as a latent representation but state that normalization is applied to tokens. Consistent terminology would improve readability. What does it mean to renormalize tokens and reuse them as context?

  3. The sample is very confusing. What does the upright index vv mean? There is a missing reference on L197. Which pretrained encoder does this work use? Why not freeze it and train the flow decoder?

  4. How is the VLM for contact transition prediction fine-tuned? How do you obtain the ground-truth labels? Why not simply use these data to train the additional head?

  5. What does “ABD datasets()” on L261 mean?

  6. Would it be possible to show an attention map for the learned sink frame? How does it support long-horizon tasks? What is the architecture of SSDD-M()? What does “decoder ()” refer to?

Reviewer_dWP1OpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies real-time interaction and long-horizon prediction in robot world models and proposes Generalizable World Simulator (GWS). Its dynamics model is trained on robot teleoperation data and combines unit-normalized latent representations, short-term context, contact-triggered long-term memory, and a streaming Transformer with a persistent attention sink and rolling KV cache. Conditioned on robot actions, the model continuously predicts multi-view observations and robot states. Evaluation includes long-horizon prediction against baselines, qualitative manipulation examples, simulated-versus-real policy evaluation, policy co-training with simulated demonstrations, and a contact-memory ablation. The results show slower visual degradation in the evaluated settings and provide preliminary evidence for policy evaluation and data-generation applications.

Critical strengths
  1. Task-motivated contact-based memory. Using contact onset and release as memory-writing events is a well-motivated way to retain scene changes caused by grasping and placement. Compared with fixed-interval sampling, this design exploits the structure of robot manipulation more directly and has potential practical value.
  2. Evaluation connects generation quality with robotic applications. Beyond long-horizon prediction metrics and qualitative demonstrations, the paper explores policy evaluation and real-world policy training using simulated demonstrations. These experiments extend the work beyond video-generation quality and provide useful preliminary evidence of its relevance to robotics.
Critical weaknesses
  1. The central claims about real-time interaction and physical fidelity are insufficiently validated. The paper reports 10 FPS while also describing a 16-step action-to-render delay in the pipeline; the relationship between throughput and interaction latency needs clarification. Moreover, PSNR, LPIPS, and FID primarily measure visual prediction quality, while the contact-memory ablation relies mainly on qualitative examples. These results do not yet establish reliable simulation of contact transitions, object persistence, and action consequences.
  2. The scale and controls of the downstream experiments limit their conclusions. Policy-evaluation correlation is based on only six mean values from two checkpoints and three tasks, with checkpoint ranking preserved on two of three tasks. The co-training experiment covers one task with ten real-world trials per policy and compares 75 real demonstrations against 150 mixed demonstrations without a matched-size real-data control. The evidence therefore remains preliminary regarding reliable policy evaluation and the relative value of simulated demonstrations.
Questions for authors
  1. What is the measured end-to-end latency from a new action input to its corresponding rendered observation under the 16-frame pipeline? How does the reported 10 FPS translate into responsiveness for teleoperation and closed-loop policies?
  2. Is there quantitative evidence across trajectories that contact-based memory preserves interaction outcomes and object states more accurately than fixed-interval memory? What are the independent contributions of unit-sphere normalization and the persistent attention sink?
  3. The reported correlation of 0.71 uses six policy–task means, and checkpoint ranking is not preserved on one task. Could you provide uncertainty estimates and explain how this ranking failure affects the simulator’s suitability for policy evaluation?
  4. How much of the co-training improvement comes from useful simulated experience versus the increase in total demonstrations? Are there controls matching dataset size or collection cost that establish the relative value of simulated demonstrations?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.