acceptodds
Withdrawn from ICLR 2027

API World Model: Agentic, Programmable, and Interactive World Model with Agent-Driven Virtual-World Rollouts

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

World models are just at a tipping point where they could really take off, with the promise of enabling agents to understand, predict, and interact with the physical world. Recent efforts have largely pursued this vision by leveraging video generation models as world simulators to predict plausible visual observations. However, these models capture the world only through its visual projections, rather than the underlying principles that govern how it operates, evolves, and responds to interventions. This fundamental gap between visual plausibility and physical consistency limits their ability to reliably simulate how the world behaves. In this paper, we introduce API World Model, a new agentic paradigm that grounds world modeling in executable programs and controllable virtual environments. Given a natural-language instruction, we first employ a coding agent to construct an interactive world in a virtual engine, where scene geometry, object states, interactions, and physical dynamics are explicitly instantiated and governed by executable programs. The world is then rendered into semantic videos and translated into photorealistic observations by a tailored video diffusion model, bridging the gap between executable physical simulation and realistic visual observations. To further support long-horizon world evolution, we devise a structured memory mechanism that maintains and updates the relevant world state across time, enabling persistent consistency across extended interactions. Moreover, we develop an agentic pipeline to construct a comprehensive benchmark, API World Bench, for evaluating world models across diverse dimensions, including spatiotemporal consistency, physical controllability, causal reasoning, and counterfactual prediction. Extensive experiments show that API World Model enables realistic, physically consistent, and controllable world rollouts, providing a promising path toward world models that capture not only how the world looks, but also how it works.

Primary area:
generative models
Keywords:
World Model, Coding Agent

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_cKiFOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

The paper proposes the API World Model, in which a coding agent (GPT-6) will build and edit a 3D world in a game engine. The game engine then renders a semantic segmentation video, then a video generation model turns it into a realistic video. The authors also proposed a World Memory module to store appearance references for each object and retrieve past frames for the background. The authors also build a benchmark containing 7 categories, and about 1k episodes. In the self-built benchmark, the proposed method scores 78.3 overall, while the best baseline SeeDance 2.5 only scores 31.4.

Critical strengths
  1. I think the high-level idea that using a game engine to handle world state and dynamics, and using video models to render appearance makes sense in terms of generating physically grounded videos. I think the split between engine and renderer is clean and gives abilities that pure video models lack. For example, in Figure 4, the objects keep moving while out of view.

  2. The benchmark tests how the world behaves, like those tasks specify explicit physical quantities, timing, and counterfactual conditions. I think this is a very important feature we need to think about when evaluating world models / video generation models.

Critical weaknesses
  1. I think the most important thing is that the current comparison is very unfair. All baselines are video models that only receive an image and a text prompt, but the proposed method has a coding agent and a physics engine. Since the benchmark tasks provide numbers, the agent can just copy them into code and the engine will execute them very precisely. I think some fair comparison are missing like using same agent and engine with direct engine rendering, or having GPT-6 rewrite the instruction into a detailed prompt for video generation models.

  2. I cannot find key details about the initial world in the paper. Each benchmark episode starts from a image made from Blender, but the method builds its world in Godot, and I couldn’t find an explanation of how the agent builds a world that matches this image. If the agent sees the Blender scene or the ground-truth layout, that is privileged information the baselines don’t have. Or, there must be an image-to-scene reconstruction step the paper doesn’t describe. Without all these details, I feel the main results are hard to interpret.

  3. The self-built benchmark is underspecified and inconsistent across the paper. Table 1 reports a Cross-View Consistency but doesn’t report Physical Quantities. However, Figure 1 and Appendix A.2.1 list Physical Quantities Control and no cross-view, so it is unclear which seven categories the benchmark actually has. The paper also doesn’t name the VLM judge and check it against human judgment.

Questions for authors

Following the weakness above:

  1. Could the authors add some fair baselines to compare? As I mentioned above, the authors can add: (1) use the same agent and engine to do direct engine rendering (without the video model); (2) use GPT-6 to rewrite the instruction into a detailed prompt, then give it to a video model like MiniMax-H3 or Seedance.

  2. In each benchmark episode, what does the agent receive? Only the image, the Blender scene, or the ground-truth layout? If the agent only gets the image, how is the Godot world built and aligned with the image?

  3. Which seven categories does the benchmark actually have? Which VLM is used as the judge? And has it been compared with human judgment?

Reviewer_r1zaOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes API World Model, a framework that separates executable world evolution from visual generation. A coding agent constructs and operates a persistent 3D environment through engine APIs, while a video diffusion model generates photorealistic observations conditioned on semantic renderings. The framework also includes entity-indexed object appearance memory and background-frame retrieval. The authors introduce API World Bench, comprising 38 task families and 1,064 evaluation episodes. Experiments report an overall score of 78.3, compared with 31.4 for the strongest listed baseline, alongside qualitative demonstrations of physical interactions, out-of-view state evolution, and appearance consistency.

Critical strengths

Clear separation of world dynamics and visual generation. Persistent engine states directly support physical parameter control, interventions, and out-of-view evolution, while semantic videos provide spatial and temporal guidance for generation. This decomposition is practically valuable for interactive simulation.

Evaluation emphasizes observable processes and measurable constraints. The benchmark examines actions, ordering, timing, and physical outcomes, and the reported system achieves substantial improvements. This evaluation direction is useful for assessing world models beyond visual plausibility.

Critical weaknesses

The experimental inputs and scoring protocol are insufficiently specified. The paper does not explain how the initial evaluation image is converted into an aligned Godot environment or whether the method accesses geometry, scene programs, or physical states used during benchmark construction. The evaluation models, numerical tolerances, and score aggregation formulas are also insufficiently detailed. These omissions limit reproducibility and make input fairness difficult to assess.

Long-term consistency and resource scalability lack sufficient validation. World Memory is evaluated through a qualitative ablation, without separately quantifying object and background memory contributions or measuring consistency across increasing interaction lengths. Background memory capacity, eviction policies, and end-to-end costs are also unclear. Consequently, the evidence does not adequately establish the benefits and costs of sustained interaction.

Questions for authors

At evaluation time, does the method receive only the same initial image and instruction as the baselines? How is the Godot scene constructed and aligned? Does this process access Blender scene programs, geometry, or physical states, and does it involve manual correction?

Please specify the evaluation models, evaluator prompts, numerical tolerances, and score aggregation formulas, and explain whether the evaluation was validated against human judgments. Why does Table 1 include cross-view consistency, whereas the appendix lists physical quantity control instead?

How are object references selected, and how are background memory capacity, updates, and eviction handled? Are quantitative results available for consistency, storage growth, and end-to-end latency across interaction lengths, as well as the separate contributions of object and background memory?

Reviewer_AWNsOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The paper uses a coding agent (GPT-6) to turn language instructions into Godot programs that build and simulate a 3D scene. The engine renders semantic videos. A ControlNet on MiniMax-H3 turns them into realistic video, and a retrieval memory keeps object and background appearance consistent. The authors also build API World Bench. Their system beats strong video generators such as Veo 3.1 and Kling 3.0.

Critical strengths
  • Using an explicit simulator for state and dynamics, and a video model only for appearance, is a sensible and practical design. It naturally supports things pure video models can't do, such as objects that keep moving while out of view.
Critical weaknesses
  • The baseline comparison is not very convincing, and some closely related work is missing. Most of the baselines are video generation models that do not use a physics simulator, so it is not surprising that the proposed method performs better on physical consistency. However, using a physics simulator or an LLM to generate simulation code is not new. GPT4Motion (Lv et al., 2024) and PhysGen (Liu et al., 2024) have explored similar ideas, but neither is discussed or compared in the paper.

  • It is unclear how much the results depend on the choice of models. The system uses GPT-6 as the coding agent and MiniMax-H3 for video generation, both of which are very strong models. However, the paper does not test other coding agents, such as open-source models, or other video generators. Without these comparisons, it is hard to tell whether the improvements come from the proposed framework or mainly from the models used.

  • Some implementation details are missing. It is unclear how the agent constructs the Godot scene from the initial image, and whether it has access to any additional information from the Blender files.

  • World Memory is not sufficiently evaluated. The paper only provides one qualitative example of World Memory, without any ablation study. It is hard to tell how much this component actually contributes to the results.

  1. Lv, J., et al. GPT4Motion: Scripting Physical Motions in Text-to-Video Generation via Blender-Oriented GPT Planning. CVPR Workshops 2024.
  2. Liu, S., Ren, Z., Gupta, S., and Wang, S. PhysGen: Rigid-Body Physics-Grounded Image-to-Video Generation. ECCV 2024.
Questions for authors
  1. What exactly does the agent receive for each episode? Is anything taken from the Blender scene files?
  2. How well does the VLM judge agree with human judgments?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.