Generalizable World Simulator: Real-Time & Long-Horizon Interaction with Diverse Objects

Interactive world models have shown great potential in serving as a simulator for robot manipulation. However, existing models are either overfit to a single environment or lack ability to simulate long-horizon contact-rich events. We present Generalizable World Simulator (GWS), a world model whose dynamics model is trained from scratch on a large robot-native teleoperation dataset, with no internet-video pretraining. By focusing on developing the optimal model architecture and learning recipe to fit robot-native data, we managed to learn a world model that is orders of magnitude smaller than SOTA video generation models, yet shows significantly better understanding of contact and causality, and runs 10 FPS on an commercial GPU. Experimentally, our model significantly outperforms previous SOTA methods across various tasks including future prediction, policy evaluation, and mixed-data co-training. We further demonstrated various potential applications where our model can be used in a robotics setting.