- Rating
- 1
- Confidence
- 4
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
The paper proposes the API World Model, in which a coding agent (GPT-6) will build and edit a 3D world in a game engine. The game engine then renders a semantic segmentation video, then a video generation model turns it into a realistic video. The authors also proposed a World Memory module to store appearance references for each object and retrieve past frames for the background. The authors also build a benchmark containing 7 categories, and about 1k episodes. In the self-built benchmark, the proposed method scores 78.3 overall, while the best baseline SeeDance 2.5 only scores 31.4.
Critical strengths
-
I think the high-level idea that using a game engine to handle world state and dynamics, and using video models to render appearance makes sense in terms of generating physically grounded videos. I think the split between engine and renderer is clean and gives abilities that pure video models lack. For example, in Figure 4, the objects keep moving while out of view.
-
The benchmark tests how the world behaves, like those tasks specify explicit physical quantities, timing, and counterfactual conditions. I think this is a very important feature we need to think about when evaluating world models / video generation models.
Critical weaknesses
-
I think the most important thing is that the current comparison is very unfair. All baselines are video models that only receive an image and a text prompt, but the proposed method has a coding agent and a physics engine. Since the benchmark tasks provide numbers, the agent can just copy them into code and the engine will execute them very precisely. I think some fair comparison are missing like using same agent and engine with direct engine rendering, or having GPT-6 rewrite the instruction into a detailed prompt for video generation models.
-
I cannot find key details about the initial world in the paper. Each benchmark episode starts from a image made from Blender, but the method builds its world in Godot, and I couldn’t find an explanation of how the agent builds a world that matches this image. If the agent sees the Blender scene or the ground-truth layout, that is privileged information the baselines don’t have. Or, there must be an image-to-scene reconstruction step the paper doesn’t describe. Without all these details, I feel the main results are hard to interpret.
-
The self-built benchmark is underspecified and inconsistent across the paper. Table 1 reports a Cross-View Consistency but doesn’t report Physical Quantities. However, Figure 1 and Appendix A.2.1 list Physical Quantities Control and no cross-view, so it is unclear which seven categories the benchmark actually has. The paper also doesn’t name the VLM judge and check it against human judgment.
Questions for authors
Following the weakness above:
-
Could the authors add some fair baselines to compare? As I mentioned above, the authors can add: (1) use the same agent and engine to do direct engine rendering (without the video model); (2) use GPT-6 to rewrite the instruction into a detailed prompt, then give it to a video model like MiniMax-H3 or Seedance.
-
In each benchmark episode, what does the agent receive? Only the image, the Blender scene, or the ground-truth layout? If the agent only gets the image, how is the Godot world built and aligned with the image?
-
Which seven categories does the benchmark actually have? Which VLM is used as the judge? And has it been compared with human judgment?