Real Where It Matters: Sim–And-Real Co-Training for Action-Conditioned World Models

Action-conditioned world models simulate future dynamics conditioned on robot actions, but their generalization is constrained by the limited coverage of real-world robot datasets. Existing datasets are dominated by successful human-teleoperated trajectories and shoulder-mounted exocentric viewpoints, leaving off-nominal interactions and broader camera viewpoints sparsely represented. Consequently, models trained on these data struggle to predict outcomes of non-optimal actions and generalize across camera viewpoints in the real world. In contrast, physics simulators can generate off-nominal interactions, such as retries and inter-object contacts, across diverse camera viewpoints at scale. In this paper, we investigate whether large-scale simulation data can complement finite real-world robot data for training action-conditioned world models for robotic manipulation. We jointly train a world model, dubbed MixWorld, on real (  95k episodes) and simulated (  550k episodes) trajectories, where real data ground predictions in the deployment domain while simulation provides substantially broader coverage of transition dynamics and camera viewpoints. Our results show that sim-and-real co-training substantially improves video prediction quality across computational metrics, model-based evaluations, and the comprehensive WorldArena benchmark. More importantly, these improvements translate into consistent gains in downstream applications, including policy evaluation and inference-time search. For policy evaluation, our world model improves Pearson correlation ($r$) by 109.3% and Spearman correlation ($\rho$) by 142.4%, while reducing Mean Maximum Rank Violation (MMRV) by 18.2% relative to the base world model. For Best-of-N inference-time search, our world model increases average success rates by 10.0%–33.3% over the base policy and 16.7%–30.0% over search with the base world model across the two tasks.