API World Model: Agentic, Programmable, and Interactive World Model with Agent-Driven Virtual-World Rollouts
Abstract
World models are just at a tipping point where they could really take off, with the promise of enabling agents to understand, predict, and interact with the physical world. Recent efforts have largely pursued this vision by leveraging video generation models as world simulators to predict plausible visual observations. However, these models capture the world only through its visual projections, rather than the underlying principles that govern how it operates, evolves, and responds to interventions. This fundamental gap between visual plausibility and physical consistency limits their ability to reliably simulate how the world behaves. In this paper, we introduce API World Model, a new agentic paradigm that grounds world modeling in executable programs and controllable virtual environments. Given a natural-language instruction, we first employ a coding agent to construct an interactive world in a virtual engine, where scene geometry, object states, interactions, and physical dynamics are explicitly instantiated and governed by executable programs. The world is then rendered into semantic videos and translated into photorealistic observations by a tailored video diffusion model, bridging the gap between executable physical simulation and realistic visual observations. To further support long-horizon world evolution, we devise a structured memory mechanism that maintains and updates the relevant world state across time, enabling persistent consistency across extended interactions. Moreover, we develop an agentic pipeline to construct a comprehensive benchmark, API World Bench, for evaluating world models across diverse dimensions, including spatiotemporal consistency, physical controllability, causal reasoning, and counterfactual prediction. Extensive experiments show that API World Model enables realistic, physically consistent, and controllable world rollouts, providing a promising path toward world models that capture not only how the world looks, but also how it works.
Then back it, or bet against it.
Related papers
Open the market on this paper to see 7 more related papers.