acceptodds
Under review as a conference paper at ICLR 2027

Code Plans, Diffusion Renders: Open-Ended Generative World Modeling

22 views

Abstract

We introduce CoDeR, a new paradigm for world modeling. Unlike existing video world models that implicitly represent world dynamics through visual observations, our system explicitly constructs an executable world with code and employs video generation models for visual realization. Specifically, we coordinate five complementary roles to translate high-level concepts into structured world rules, executable dynamics, and perceptual observations. This design enables long-term memory, open-ended interactions, autonomous world evolution, and multi-agent scenarios, where multiple entities can act, interact, and evolve persistently beyond the current observation. Extensive experiments demonstrate that our framework substantially extends the capabilities of existing world models, enabling long-term memory, open-ended interactions, autonomous evolution, and persistent multi-agent dynamics, while achieving state-of-the-art performance across multiple evaluation settings. Code and model weights will be made publicly available.

Primary area:
generative models
Keywords:
world model, video generation

Reviews

Among:
Shape:
1234this paper, Top 17%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.75 over 4 reviews, above 83% of the others.
Reviewer_Hz9bOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

CoDeR constructs executable worlds using LLM-based coding agents and employs a video model to synthesize their visual observations. Logical Spaces organize world construction through functional modules and interface contracts. Observation as World Registration projects generated appearance back onto world surfaces for subsequent observations, while Visual Cue Hacking adapts the video model to whitebox inputs. Validation includes WorldScore comparisons, component ablations, a preliminary world-construction experiment, qualitative interaction and evolution examples, and a user study. The results support the system’s feasibility and improved generation performance, although broader claims about persistent dynamic consistency remain less directly evaluated.

Critical strengths
  1. CoDeR connects automated world construction, executable state transitions, generative rendering, and persistent appearance registration into a functioning system. This enables rules and action consequences to persist independently of the current Traveler’s view, while generated appearance can be reused across observations.
  2. Table 1 reports the highest average WorldScore among the evaluated methods, and Table 2 shows improvements from registration and Visual Cue Hacking. Appendix A.3 additionally reports reduced construction time under a controlled comparison. Together, these results support the system’s usefulness beyond qualitative demonstrations.
Critical weaknesses

1.The experiments do not sufficiently measure whether generated observations faithfully represent executable state, particularly after objects move, disappear, or change. Visual-quality metrics provide only indirect evidence for persistent identities and event consistency. Appendix A.4 also specifies ordered registration and auxiliary viewpoints for changing regions, which narrow the demonstrated setting and should be reflected more clearly in the main claims. 2. The actual evaluation sample counts, scene and interaction coverage, and handling of failures, retries, and candidate selection are insufficiently specified. Appendix A.3 evaluates only two generations per condition on one construction task; its workflow checks do not establish success rates across independent worlds or diverse interactions. These omissions limit confidence in the broader reliability claims. Missing end-to-end timing, including Artist inference and auxiliary observations, further limits assessment of practical applicability.

Questions for authors

Q1. Tables 1 and 2 report positive quantitative results, but the evaluation coverage and success accounting remain unclear. How many examples were evaluated, which scene and interaction categories were covered, and which videos were scored using the selected VBench metrics? How were reconstruction failures, retries, manual corrections, and candidate selection handled? Beyond the two generations per condition in Appendix A.3, are attempt and success counts available for the demonstrated interaction tasks? Q2. Correct executable state does not automatically guarantee that generated RGB faithfully represents that state. What direct evidence establishes consistency between world state and generated observations, particularly for persistent object identities, event outcomes, and changes during observation gaps? Appendix A.4 specifies ordered registration and auxiliary viewpoints for changing regions. Which reported demonstrations depend on these viewpoints, and how is outdated registered appearance handled when objects move, disappear, or change state? Q3. Appendix A.3 reports a reduction in average world-construction time from 1703.7 to 992.0 seconds, but this does not characterize the complete pipeline. For a representative evaluated setting, what are the separate costs of world construction, Artist inference, and appearance registration or updates? Please specify the hardware, resolution, generated frame count, and whether auxiliary observations are included. This question concerns the practical cost of the current offline framework, rather than requiring real-time operation.

4 comments

Dear Reviewer Hz9b,

we sincerely thank you for your recognition of the soundness of our method design and our experimental performance. Below is our detailed response to your constructive comments.

W1/Q2: Consistency between Generated Videos and World States

The generated videos highly match the corresponding world states. To evaluate whether the Artists generate videos that faithfully follow their whitebox inputs, we constructed 30 whitebox worlds spanning indoor and outdoor environments, including bedrooms and cities, as well as more imaginative settings such as underwater environments and alien planets. These worlds feature dynamics ranging from rigid-body to non-rigid motion, including doors opening and closing, vehicles driving, robotic arms manipulating objects, and people walking.
For each world, we sampled a camera trajectory spanning 248 frames and rendered the corresponding whitebox video, producing 30 whitebox videos in total. We then used the Artists to render each video with prompts specifying three different visual styles, yielding 90 videos for evaluation. Our evaluation comprises two phases: (1) alignment between generated videos and whitebox inputs, and (2) retention and updating of registered appearance.
For the first phase, we estimate the depth of the generated videos using Video Depth Anything and evaluate their spatial alignment with the whitebox world using AbsRel and δ₁. The results are reported below. We also visualize examples of relative depth errors in Fig. 15 in the revised pdf.

EvaluationAbsRel ↓δ₁ ↑
Phase 10.173284.1%

Depth evaluation alone does not establish the consistency of registered appearance across time, viewpoints, and dynamic conditions. We therefore conduct a second evaluation. We register the previously generated videos into the world, then construct new observation trajectories by retaining the camera positions along the original trajectories while varying the camera orientations. We generate videos along these new trajectories and compute SSIM and PSNR against the original videos within the registered regions.
We evaluate two settings: using the original text prompt and using an empty text prompt, with the registered appearance serving as the sole style cue in the latter setting. The results are reported below. Appearance is registered onto the corresponding 3D mesh surfaces, allowing it to remain associated with objects as they move.

EvaluationSSIM ↑PSNR ↑
Phase 2 (w/o prompt)0.883227.18
Phase 2 (w/ prompt)0.946130.39

W2.1/Q1: Evaluation Coverage

We have expanded the evaluation across tasks and backbone models, and clarify the sampling, success accounting, and computational costs below. Tables 1 and 2 are evaluated on a stratified subset of 100 WorldScore examples, comprising 60 static examples across all 10 scene categories and 40 dynamic examples across all five motion categories. Each category contains equal numbers of photorealistic and stylized examples. We also extend the Logical Spaces study at three levels of world complexity:

  • Simple: Static worlds with basic functionality, such as navigation. This level includes two scenarios: a bedroom and a small park.
  • Medium: Worlds with richer functionality, dynamics, and internal logic. This level includes two scenarios: a house with a living room, multiple rooms, and interactions such as opening and closing doors and controlling a television with a remote; and an FPS game with shooting mechanics, weapon switching, and health management.
  • Complex: Worlds integrating multiple functional systems. We reuse the dense-town task described in Appendix A.3.

Each world-construction task is independently repeated three times. To further assess robustness across backbone models, we conduct ablations with and without Logical Spaces using GPT-6 Luna with high reasoning effort and GPT-6 Astra with medium reasoning effort. The results are presented below. (Values are reported as Default → w/ Logical Spaces.)

ModelLevelSuccess ↑Journeys ↑avg Time (s) ↓avg Tokens (k) ↓
LunaSimple6/6 → 6/618/18 → 18/18225.9 → 206.728.06 → 22.34
LunaMedium4/6 → 5/618/33 → 26/33704.5 → 421.843.62 → 28.71
LunaComplex0/3 → 0/30/21 → 0/21timeout → timeout50.14 → 56.90
AstraSimple6/6 → 6/618/18 → 18/18117.2 → 94.78.73 → 8.39
AstraMedium6/6 → 6/632/33 → 33/33585.0 → 301.432.88 → 20.82
AstraComplex3/3 → 3/318/21 → 21/211638.6 → 941.552.91 → 32.68

W2.2/Q3: End-to-end Latency

Our architecture is fully orthogonal to existing video diffusion acceleration techniques, and the additional runtime overhead introduced by our design is on the order of milliseconds.

The latency introduced by our framework mainly comes from two stages: whitebox world initialization and the Observation as World Registration mechanism.

  • Whitebox World Initialization: As reported in the table above, the one-time initialization time is primarily driven by the synthesis of complex cross-system logic; for simple roaming tasks that only require basic scene geometry, initialization takes only 30 seconds to 2 minutes.
  • Observation & World Registration Overhead: For Observation (rendering the whitebox and depth videos and encoding them into latents), the latency is near real-time at 56 ms per 22 frames on a single NVIDIA H100 GPU. For World Registration (decoding generated latents back to RGB and registering them onto world surfaces), the measured latency is 61 ms per 22 frames on a single H100 GPU.
  • Denoising Generation Latency: Equipped with DMAD (few-step LoRA) under a 4-step inference setting on 32 H100 GPUs, our Artist model generates 124 frames in 6.76 s and 22 frames in 0.92 s, corresponding to ~18 FPS and ~22–24 FPS, respectively. All tests above are conducted at a resolution of 1280 x 704.

Thank you for the clarifications and additional experiments. Most of my main questions have now been addressed. In particular, the evaluation sample counts, expanded world-construction study, and runtime breakdown resolve several reporting gaps identified in my initial review and strengthen the paper’s empirical support.

Thank you for carefully reviewing our responses and additional experiments. We are pleased that most of your main questions have been addressed, and we appreciate your recognition of the strengthened empirical support.

Thank you again for your time and thoughtful assessment.

Reviewer_ADetOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces CoDeR, a framework that uses LLMs and coding agents to construct a 3D whitebox world with explicit states and executable rules, while a video generation model synthesizes visual observations. By combining modular world construction with the registration of generated appearance back into the world, the framework supports cross-view memory, out-of-sight state evolution, and a shared world across multiple observers.

Critical strengths
  1. Explicit separation of world state and visual observations. The framework maintains hidden attributes, interaction rules, and out-of-sight events through executable programs. This provides a practically valuable approach to improving state consistency and controllability over purely video-based world models.

  2. Demonstrations across multiple capabilities, supported by component evaluations. The paper demonstrates viewpoint revisitation, extensible interactions, out-of-sight evolution, and information sharing across observers, alongside quantitative evaluations, a user study, and component ablations. In particular, the ablations of Observation as World Registration and Visual Cue Hacking provide evidence supporting their effectiveness.

Critical weaknesses
  1. The incremental novelty over closely related work is not sufficiently established. The central idea of separating explicit world state from generative rendering has already been explored in Magpie, Marionette, and MASS. The appearance back-projection and cross-view reuse underlying Observation as World Registration also have clear connections to prior generative texturing and 3D memory methods. The potential contributions therefore lie primarily in multi-agent world construction, appearance registration onto whitebox surfaces, and their integration into a unified system. However, the paper lacks an in-depth comparison with the closest approaches, and the limited scale of the Logical Spaces ablation provides insufficient evidence to support the stronger claim of a new world-modeling paradigm.

  2. The framework’s expressive scope is constrained by the code-generated whitebox and its simulation capabilities. Its applicability may be limited by the underlying geometric representations, interaction rules, and dynamics modeling. Although the appendix acknowledges failures involving complex hand–object interactions and object interpenetration, the paper does not systematically evaluate dense dynamic scenes, non-rigid motion, or events that are not explicitly encoded. It also remains unclear whether the video model’s learned dynamics priors can compensate for limitations in the underlying world model. Consequently, the trade-off between improved controllability and the ability to represent open-ended scenes is insufficiently characterized. For example, in a dense urban environment with many moving pedestrians, the scalability of whitebox construction and state maintenance remains unverified. If pedestrians are explicitly modeled in code, the construction cost and interaction accuracy at scale should be evaluated. If they are introduced by the Artist, the paper should explain how they acquire persistent identities and interactive states while maintaining consistency across viewpoints and time.

  3. The current framework does not support fully real-time interaction, limiting its practical applicability as an interactive world model. As acknowledged in Appendix A.10, the staged pipeline of constructing a whitebox world and subsequently generating visual observations introduces additional latency. However, the paper does not adequately characterize initialization costs, end-to-end latency from an action to visual feedback, or throughput under multiple observers. The demonstrations therefore do not establish the feasibility of sustained, low-latency online interaction.

Questions for authors
  1. Figure 5 demonstrates a continuously rising water level during pouring, but how does the framework handle more complex dynamics, such as overflow, splashing, and water pooling on the floor? Which changes must be explicitly programmed by the Executors, and which can be generated autonomously by the Artist? If the Artist generates an event absent from the whitebox representation, can the system incorporate it into the world state so that it affects subsequent interactions, rather than merely registering its visual appearance?

  2. In which scenarios does CoDeR offer clear advantages over an alternative pipeline that generates a complete, editable 3D scene—for example, using World Labs’ Marble—and combines it with coding agents, a simulator, and a conventional renderer? In particular, once the underlying world state is maintained in code, what measurable benefits does chunk-wise video generation provide over directly rendering 3D assets, and what additional latency and consistency costs does it introduce?

  3. How does the framework handle viewpoints or exploration beyond the constructed whitebox world? If the camera faces a region without defined geometry or state, does the system restrict camera movement, allow the Artist to synthesize visual content, or trigger incremental world construction by the Executors? If incremental expansion is supported, how are geometric, state, and visual consistency maintained between existing and newly constructed regions, and how is the additional latency controlled? Please clarify whether “open-ended” includes online exploration beyond the initial world boundaries.

8 comments

Dear Reviewer ADet,

Thank you for recognizing the advantages of our approach over video-based world models and for your positive assessment of our qualitative and quantitative experiments. Below, we respond in detail to your insightful comments.

W1.1: Novelty and Comparisom with Closely Related Work

CoDeR contributes a novel agentic framework for collaborative construction of executable worlds and registration of generated observation. The closest approaches differ in how worlds are constructed and how generated appearance is retained:

  • Magpie operates on designer-authored gameplay and maintains visual continuity through anchor frames, viewpoint-retrieved observations, and recent chunks. CoDeR additionally addresses coding agents construction and surface-registered appearance reuse. Magpie itself identifies persistent 3D visual memory as a limitation and future direction.
  • Marionette learns articulated-state dynamics and renders them through a deterministic geometric bridge and a video observation model. CoDeR generates executable rules and modules through coding agents. Its registration mechanism also differs from Marionette’s first-frame and chunk-relay appearance conditioning.
  • MASS learns authoritative state transitions under a declared game schema and synchronizes player views. Its multiplayer coordination concerns world execution; Logical Spaces addresses coordination among agents constructing the world, through functional decomposition, interface contracts, state ownership, and runtime bindings.

It is also worth noting that Magpie, Marionette, and MASS appeared as contemporaneous preprints in August 2026, shortly before the submission deadline. The ICLR 2027 guidelines state that comparisons with contemporaneous or unpublished arXiv work are not required. Nevertheless, we appreciate the opportunity to clarify the technical relationships and specific contributions of our approach.

W1.2: More Ablation on Logical Spaces

Logical Spaces provides consistent improvements across different scene settings. We evaluate its effectiveness at three levels of world complexity:

  • Simple: Static worlds with basic functionality, such as navigation. This level includes two scenarios: a bedroom and a small park.
  • Medium: Worlds with richer functionality, dynamics, and internal logic. This level includes two scenarios: a house with a living room, multiple rooms, and interactions such as opening and closing doors and controlling a television with a remote; and an FPS game with shooting mechanics, weapon switching, and health management.
  • Complex: Worlds integrating multiple functional systems. We reuse the dense-town task described in Appendix A.3.

Each world-construction task is independently repeated three times. To further assess robustness across backbone models, we conduct ablations with and without Logical Spaces using GPT-6 Luna with high reasoning effort and GPT-6 Astra with medium reasoning effort. The results are presented below. (Values are reported as Default → w/ Logical Spaces.)

ModelLevelSuccess ↑Journeys ↑avg Time (s) ↓avg Tokens (k) ↓
LunaSimple6/6 → 6/618/18 → 18/18225.9 → 206.728.06 → 22.34
LunaMedium4/6 → 5/618/33 → 26/33704.5 → 421.843.62 → 28.71
LunaComplex0/3 → 0/30/21 → 0/21timeout → timeout50.14 → 56.90
AstraSimple6/6 → 6/618/18 → 18/18117.2 → 94.78.73 → 8.39
AstraMedium6/6 → 6/632/33 → 33/33585.0 → 301.432.88 → 20.82
AstraComplex3/3 → 3/318/21 → 21/211638.6 → 941.552.91 → 32.68

W2: Expressive scope between Code and Diffusion

The expressiveness of code is reflected in intrinsic logic such as geometric layout and interactions, while the expressiveness of diffusion is reflected in generative capabilities such as visual details and dynamic compensation. Together, the two jointly build the overall expressiveness of the world model.
Appendix A.9 indicates that the coding agent has difficulty modeling complex interactions and non-rigid motion, but our experiments find that the video model can compensate for this deficiency; please see our supplementary visualization, Fig. 17, in the revised pdf.
We show three scenarios: (a) the dense-urban with pedestrians that you mentioned; (b) highly dynamic fire; (c) tool use (rigging).
As shown in Fig. 17 (a), although the coding agent only used coarse geometry to represent trees and an SMPL model to represent people, diffusion can still complete plausible visual details (leaves, clothing, etc.). In addition, the walking motion of the human body maintained by the coding agent is relatively mechanical, and the video model can also compensate for motion smoothness.
Fig. 17 (b) demonstrates the video model's high dynamic compensation: even if the coding agent only used a few coarse cones to represent the campfire, the video model can still translate them into dynamic flames, rather than rendering hard cones in a copy-paste manner. In addition, the reflection of the flames in the house glass in Fig. 17 (b) and the reflection of the human body on the floor in Fig. 17 (c) also demonstrate that diffusion can compensate for the expressiveness that the coding agent lacks or finds difficult to achieve, even when it is not explicitly encoded.
Fig. 17 (c) demonstrates the advantage of our method as an agentic framework in that it can call on external mature tools to compensate for the coding model's own limited capabilities, and that the video model can accurately capture and compensate for the corresponding expressiveness. In other words, purely relying on a coding agent to turn an SMPL human body model into an anime-style character requires a great deal of effort, such as drawing and stitching garment panels and modeling facial features, whereas artists can directly transfer powerful generative capabilities to such whitebox scenarios thanks to our Visual Cue Hacking strategy.

W3: End-to-end Latency

Our architecture is fully orthogonal to existing video diffusion acceleration techniques, and the additional runtime overhead introduced by our design is on the order of milliseconds.

The latency introduced by our framework mainly comes from two stages: whitebox world initialization and the Observation as World Registration mechanism.

  • Whitebox World Initialization: As reported in the table above, the one-time initialization time is primarily driven by the synthesis of complex cross-system logic; for simple roaming tasks that only require basic scene geometry, initialization takes only 30 seconds to 2 minutes.
  • Observation & World Registration Overhead: For Observation (rendering the whitebox and depth videos and encoding them into latents), the latency is near real-time at 56 ms per 22 frames on a single NVIDIA H100 GPU. For World Registration (decoding generated latents back to RGB and registering them onto world surfaces), the measured latency is 61 ms per 22 frames on a single H100 GPU.
  • Denoising Generation Latency: Equipped with DMAD (few-step LoRA) under a 4-step inference setting on 32 H100 GPUs, our Artist model generates 124 frames in 6.76 s and 22 frames in 0.92 s, corresponding to ~18 FPS and ~22–24 FPS, respectively. All tests above are conducted at a resolution of 1280 x 704.

Q1: Executable Dynamics and Generated Visual Effects

Executors define persistent state changes and their interactive consequences, while the Artist supplies their detailed visual realization. Supporting overflow and floor pooling as interactive events would require corresponding state variables and transition rules, such as container capacity, overflow conditions, and accumulated water on the floor. These can use simplified representations without simulating every droplet. The Artists supplement these representations with visual details, such as ripples and splash appearance. Our additional flame example in Fig. 17 (b) illustrates this capacity for visual enrichment: coarse cones condition the generation of dynamically varying flames. Such examples demonstrate visual completion beyond the coarse representation; they do not establish the recovery of missing physical rules.
In the current framework, an event generated only by the Artist is not automatically converted into executable world state. Observation as World Registration associates generated appearance with existing surfaces for subsequent reuse.

Q2: Advantages of using a Video Model

As discussed in W2, using diffusion can bring benefits such as visual compensation and dynamic modeling, compared with explicit 3D modeling. A critical limitation of World Labs Marble is its modeling of dynamics, since Marble’s demos are almost all static 3D Gaussian/mesh scenes, which hinders flexibility and diversity. In addition, we also used GPT-6 Astra to construct a well-textured 3D scene with color and lighting purely in Three.js; the result is shown in Fig. 16. Compared with the paradigm that introduces diffusion, traditional rendering is costly, and the upper limit of expressiveness is constrained by the graphics engine, whereas introducing diffusion can remedy this shortcoming, both efficiently and qualitatively.

Q3: Viewpoints beyond the Constructed Whitebox World

When the viewpoint moves into a region where no whitebox has been constructed, the depth-map input is used as the primary guidance. In addition, our framework supports online exploration. Because the Artist model takes both the whitebox and the depth map as inputs, when the viewpoint reaches an area that has not yet been whiteboxed, depth is used as visual guidance. Furthermore, at the initialization stage (described in W3), extending the world’s spatial span in terms of scale is not difficult; it can be achieved by directly duplicating neighborhood blocks. For more heterogeneous regions, online world expansion appends validated modules while preserving existing code and registered runtime state, with rollback when integration violates these constraints. During incremental expansion, existing module files remain unchanged, and new requirements are implemented through additional modules where the existing interfaces support them. These interfaces remain callable: new modules read copies of shared outputs and may invoke exposed commands after attachment, while each existing module retains exclusive write ownership of its registered state. Interface validation and attachment checks reject incompatible additions and roll back changes to existing registered state, providing explicit safeguards for integrating new functionality into the running world.

Reviewer_Gmm2OpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
3(+1)
Read review
Paper summary

The paper proposes CoDeR, a world-modeling pipeline that keeps the world state in code and uses a video model to handle appearance. The authors show qualitative examples suggesting that the system supports long-term memory, open-ended interaction, off-screen evolution and multi-agent sharing.

Critical strengths

The paper uses code to model an explicit world state. This lets the system support memory, off-screen changes and multi-agent sharing, which are known weak points of pixel-based video world models. The two proposed methods, Logical Spaces and Observation as World Registration, are both reasonable ideas: the first helps organize world construction, and the second keeps the rendered appearance consistent.

Critical weaknesses

First, the comparison is not very fair. All baselines are open-source video world models. Beating them only shows that GPT-6 Astra has strong world knowledge and that the engine it writes maintains the state well. Second, the novelty is limited. Most of the capabilities claimed in the paper are properties that code or 3D engines already have. In my view, the genuinely new parts are Logical Spaces and the registration mechanism, but the paper does not discuss them enough. It does not make clear how Logical Spaces helps multiple agents build a large world and keep it consistent, how registration maintains appearance consistency, or how that consistency should be evaluated.

Questions for authors

1.Why does the coding agent have to output an untextured whitebox? GPT-6 Astra can already write Three.js scenes with materials, textures and lighting that look quite good. If the world were colored at construction time, the video model would only need to do style enhancement. There would be no need to fine-tune the Artist for coloring, and no need for the registration mechanism. What does the whitebox design actually buy you?

2.What does the Artist do beyond stylization? Does it provide any information that is not already in the code world?

3.How much do the results depend on GPT-6 Astra? The largest baseline is only around 33B. Could you use a smaller coding model, such as GPT-6 Luna, or even an open-source MoE model such as Qwen3-Coder-480B-A35B or Qwen3-235B-A22B, for the coding part?

4.What is the end-to-end latency? Most of the baselines are real-time streaming models.

5.I see Logical Spaces as the main point of this paper. Building a Three.js scene with code is not hard in itself. What is interesting is how to keep a much larger scene consistent. Could you show worlds at a larger scale, for example a town or a city, and explain how the Executors keep the world consistent with each other?

6.I understand the authors may use language models to polish the writing, but please check for obvious signs, such as overuse of “rather than” and a lot of defensive phrasing. For example, the last paragraph on page 4 reads: “Representing the world through executable code, rather than raw pixels, allows the hidden information to be explicitly encoded and enforced, rather than being implicitly and unreliably inferred from visual appearance.” This one sentence contains two “rather than”s, and a reader who has gone through the earlier text would not misread it anyway.

5 comments

Dear Reviewer Gmm2,

Thank you for recognizing our framework's ability to mitigate key shortcomings of pixel-based video world models, as well as the value of Logical Spaces and Observation as World Registration. Below, we provide detailed responses to your constructive comments and questions.

W1/W2.1: Comparison Fairness and Novelty

CoDeR is a novel agentic world modeling framework that coordinates multiple specialized roles. Its performance reflects their combined contributions and cannot be attributed solely to GPT-6 Astra's world knowledge. Specifically, bridging coding agents and video models into a coherent world system requires solving three non-trivial challenges beyond GPT-6 Astra's inherent capabilities:

  1. How should the state maintained by GPT-6 Astra be observed?

While GPT-6 can directly generate well-textured Three.js scenes (as also discussed in Q1 below), directly observing such code-rendered states hits a strict expressiveness ceiling locked by the underlying graphics engine and incurs heavy token and construction costs. Combining GPT-6 with a video model overcomes this expressiveness bottleneck while allowing the coding agents to represent world states efficiently as lightweight whiteboxes (please see Fig. 16 in the revised pdf). Moreover, connecting symbolic code states to video diffusion is non-trivial due to the modality gap: directly conditioning on whitebox renders yields rigid, toy-like visuals. We therefore introduce Visual Cue Hacking to bridge whitebox states and generative rendering, whose necessity is verified both qualitatively (Fig. 9) and quantitatively (Tab. 2).

  1. How can coding agents efficiently construct systematic, complex worlds?

Though GPT-6 has world knowledge, using it to build a large-scale, multi-system world is non-trivial due to context-window bottlenecks, cross-module conflicts, and complex dynamic modeling failures (as analyzed in Appendix A.9). To expand the boundary of world construction, we design a five-role agentic hierarchy together with Logical Spaces. Within this agentic harness, Executors also invoke external specialized tools to compensate for GPT-6 Astra’s intrinsic limitations—for instance, calling rigging and motion tools (e.g., SMPL-H) to model complex articulated human dynamics that raw LLM code generation cannot handle directly (as shown in Fig. 17(c)).

  1. Moving from naive Code → Video rendering to bidirectional Code ↔ Video interaction

While sota video models can zero-shot synthesize visuals from an input world state, treating the code state merely as a one-off "visual prompt" (Code → Video) causes severe appearance drift whenever a viewpoint is revisited. A central challenge in world modeling is how to feed the synthesized visual appearance back into the underlying world state. We propose Observation as World Registration to establish a bidirectional Code ↔ Video feedback loop. This closed-loop paradigm enables long-term visual memory and even multi-Traveler observation-interaction consistency.

W2.2: Technical Details of Logical Spaces and World Registration

Logical Spaces decompose a world into functional modules with explicit state ownership and input/output contracts. Multiple agents build these modules in parallel, coordinating dependencies through shared interface declarations and validating published modules before integration. A unified runtime enforces registered-state ownership, checks interface compatibility, and advances all modules under a shared clock. World expansion appends validated modules while preserving existing code and registered runtime state, with rollback when integration violates these constraints. Registration mechanism back project generated frames to the whitebox 3D mesh surface based on corresponding depth. We also design a two-phase experiment to measure this consistency. We constructed 30 whitebox worlds spanning indoor and outdoor environments, including bedrooms and cities, as well as more imaginative settings such as underwater environments and alien planets. These worlds feature dynamics ranging from rigid-body to non-rigid motion, including doors opening and closing, vehicles driving, robotic arms manipulating objects, and people walking. For each world, we sampled a camera trajectory spanning 248 frames and rendered the corresponding whitebox video, producing 30 whitebox videos in total. We then used the Artists to render each video with prompts specifying three different visual styles, yielding 90 videos for evaluation. Our evaluation comprises two phases: (1) alignment between generated videos and whitebox inputs, and (2) retention and updating of registered appearance. For the first phase, we estimate the depth of the generated videos using Video Depth Anything and evaluate their spatial alignment with the whitebox world using AbsRel and δ₁. The results are reported below. We also visualize examples of relative depth errors in Fig. 15 in the revised pdf.

EvaluationAbsRel ↓δ₁ ↑
Phase 10.173284.1%

We then conduct a second phase. We register the previously generated videos into the world, then construct new observation trajectories by retaining the camera positions along the original trajectories while varying the camera orientations. We generate videos along these new trajectories and compute SSIM and PSNR against the original videos within the registered regions.

EvaluationSSIM ↑PSNR ↑
Phase 2 (w/o prompt)0.883227.18
Phase 2 (w/ prompt)0.946130.39

Q1: Why Whitebox?

With a diffusion renderer, the whitebox serves as an efficient and intuitive representation, while our framework also natively supports pre-textured geometry by treating initial textures as part of the registered world.
As explained in W1/W2.1, relying solely on coding agents to build and directly observe Three.js scenes is suboptimal in both visual quality and token/time efficiency, making the diffusion Artist essential. Under this paradigm, using an untextured whitebox instead of code-authored textures lets coding agents focus purely on geometry and interaction logic while allowing the same world to be flexibly rendered into diverse visual styles. Finally, our model is compatible with well-textured geometry as input, where existing textures are naturally treated as pre-registered observations.

Q2: Capability of Artists

Our experiment proves that Artists can compensate for the shortcomings of coding agents, rather than just serving as stylicizers.

As shown in Fig. 17 (a), although the coding agent only used coarse geometry to represent trees and an SMPL model to represent people, Artists can still complete plausible visual details (leaves, clothing, etc.). In addition, the walking motion of the human body maintained by the coding agent is relatively mechanical, and the Artists can also compensate for motion smoothness. Fig. 17 (b) demonstrates the video model's high dynamic compensation: even if the coding agent only used a few coarse cones to represent the campfire, the video model can still translate them into dynamic flames, rather than rendering hard cones in a copy-paste manner. In addition, the reflection of the flames in the house glass in Fig. 17 (b) and the reflection of the human body on the floor in Fig. 17 (c) also demonstrate that diffusion can compensate for the expressiveness that the coding agent lacks or finds difficult to achieve, even when it is not explicitly encoded.

Q3: Performance using Luna

Equipped with our proposed Logical Spaces strategy, a smaller coding model also achieves strong performance.

We evaluate the performance of a smaller coding model (GPT-6 Luna) across three levels of world complexity:

  • Simple: Static worlds with basic functionality, such as navigation. This level includes two scenarios: a bedroom and a small park.
  • Medium: Worlds with richer functionality, dynamics, and internal logic. This level includes two scenarios: a house with a living room, multiple rooms, and interactions such as opening and closing doors and controlling a television with a remote; and an FPS game with shooting mechanics, weapon switching, and health management.
  • Complex: Worlds integrating multiple functional systems. We reuse the dense-town task described in Appendix A.3.

Each world-construction task is independently repeated three times. The results are presented below. (Values are reported as Default → w/ Logical Spaces.)

ModelLevelSuccess ↑Journeys ↑avg Time (s) ↓avg Tokens (k) ↓
LunaSimple6/6 → 6/618/18 → 18/18225.9 → 206.728.06 → 22.34
LunaMedium4/6 → 5/618/33 → 26/33704.5 → 421.843.62 → 28.71
LunaComplex0/3 → 0/30/21 → 0/21timeout → timeout50.14 → 56.90
AstraSimple6/6 → 6/618/18 → 18/18117.2 → 94.78.73 → 8.39
AstraMedium6/6 → 6/632/33 → 33/33585.0 → 301.432.88 → 20.82
AstraComplex3/3 → 3/318/21 → 21/211638.6 → 941.552.91 → 32.68

Q4: End-to-end Latency

Our architecture is fully orthogonal to existing video diffusion acceleration techniques, and the additional runtime overhead introduced by our design is on the order of milliseconds.

The latency introduced by our framework mainly comes from two stages: whitebox world initialization and the Observation as World Registration mechanism.

  • Whitebox World Initialization: As reported in the table above, the one-time initialization time is primarily driven by the synthesis of complex cross-system logic; for simple roaming tasks that only require basic scene geometry, initialization takes only 30 seconds to 2 minutes.
  • Observation & World Registration Overhead: For Observation (rendering the whitebox and depth videos and encoding them into latents), the latency is near real-time at 56 ms per 22 frames on a single NVIDIA H100 GPU. For World Registration (decoding generated latents back to RGB and registering them onto world surfaces), the measured latency is 61 ms per 22 frames on a single H100 GPU.
  • Denoising Generation Latency: Equipped with DMAD (few-step LoRA) under a 4-step inference setting on 32 H100 GPUs, our Artist model generates 124 frames in 6.76 s and 22 frames in 0.92 s, corresponding to ~18 FPS and ~22–24 FPS, respectively. All tests above are conducted at a resolution of 1280 x 704.

Q5: Building Larger Worlds

Geometric extent can be increased by repeating or instancing blocks, and Logical Spaces address the additional challenge of expanding interacting functionality. Each functional system is implemented in a separate spaces .mjs file declaring its owned state and required/provided interfaces, allowing Executors to develop modules in parallel. During incremental expansion, existing module files remain unchanged, and new requirements are implemented through additional modules where the existing interfaces support them. These interfaces remain callable: new modules read copies of shared outputs and may invoke exposed commands after attachment, while each existing module retains exclusive write ownership of its registered state. Interface validation and attachment checks reject incompatible additions and roll back changes to existing registered state, providing explicit safeguards for integrating new functionality into the running world.

Q6: About Polishing

We sincerely thank you for your suggestions on improving the quality of our writing. We have carefully revised the manuscript and marked the revised sentences in red.

Reviewer_Gj4wOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

CoDeR uses LLM agents to build executable Three.js worlds and a video model to render them. Generated appearance is registered back into the world for reuse. The paper proposes modular construction through Logical Spaces and renderer training through Visual Cue Hacking, evaluated with benchmarks, ablations, a user study, and interaction examples.

Critical strengths
  1. Code maintains world dynamics, while appearance registration lets later views reuse generated content. The design connects persistent simulation with video generation.
  2. CoDeR leads nine of ten WorldScore metrics among the compared methods. Ablations support both rendering components, and user ratings favor the system.
Critical weaknesses
  1. Examples and user ratings support interaction consistency, but the paper does not report a direct measurement of how often rendered outcomes match executable state across views and time.
  2. Logical Spaces is tested on one task with two runs per condition. The reported speedup is promising, but its stability is unclear; 14/14 versus 13/14 passed workflows provides limited evidence of improved reliability.
Questions for authors
  1. Could you report the individual runtimes in Table 3? If available, do results from other tasks or seeds show a similar speedup?
2 comments

Dear Reviewer Gj4w,

Thank you for your recognition of our method design and experimental results. Below are our detailed responses to your valuable comments.

W1: Consistency between Generated Videos and World States

The generated videos highly match the corresponding world states. To evaluate whether the Artists generate videos that faithfully follow their whitebox inputs, we constructed 30 whitebox worlds spanning indoor and outdoor environments, including bedrooms and cities, as well as more imaginative settings such as underwater environments and alien planets. These worlds feature dynamics ranging from rigid-body to non-rigid motion, including doors opening and closing, vehicles driving, robotic arms manipulating objects, and people walking.
For each world, we sampled a camera trajectory spanning 248 frames and rendered the corresponding whitebox video, producing 30 whitebox videos in total. We then used the Artists to render each video with prompts specifying three different visual styles, yielding 90 videos for evaluation. Our evaluation comprises two phases: (1) alignment between generated videos and whitebox inputs, and (2) retention and updating of registered appearance.
For the first phase, we estimate the depth of the generated videos using Video Depth Anything and evaluate their spatial alignment with the whitebox world using AbsRel and δ₁. The results are reported below. We also visualize examples of relative depth errors in Fig. 15 in the revised pdf.

EvaluationAbsRel ↓δ₁ ↑
Phase 10.173284.1%

Depth evaluation alone does not establish the consistency of registered appearance across time, viewpoints, and dynamic conditions. We therefore conduct a second evaluation. We register the previously generated videos into the world, then construct new observation trajectories by retaining the camera positions along the original trajectories while varying the camera orientations. We generate videos along these new trajectories and compute SSIM and PSNR against the original videos within the registered regions.
We evaluate two settings: using the original text prompt and using an empty text prompt, with the registered appearance serving as the sole style cue in the latter setting. The results are reported below. Appearance is registered onto the corresponding 3D mesh surfaces, allowing it to remain associated with objects as they move.

EvaluationSSIM ↑PSNR ↑
Phase 2 (w/o prompt)0.883227.18
Phase 2 (w/ prompt)0.946130.39

W2: Robustness of Logical Spaces

Logical Spaces provides consistent improvements across different scene settings. We evaluate its effectiveness at three levels of world complexity:

  • Simple: Static worlds with basic functionality, such as navigation. This level includes two scenarios: a bedroom and a small park.
  • Medium: Worlds with richer functionality, dynamics, and internal logic. This level includes two scenarios: a house with a living room, multiple rooms, and interactions such as opening and closing doors and controlling a television with a remote; and an FPS game with shooting mechanics, weapon switching, and health management.
  • Complex: Worlds integrating multiple functional systems. We reuse the dense-town task described in Appendix A.3.

Each world-construction task is independently repeated three times. To further assess robustness across backbone models, we conduct ablations with and without Logical Spaces using GPT-6 Luna with high reasoning effort and GPT-6 Astra with medium reasoning effort. The results are presented below. (Values are reported as Default → w/ Logical Spaces.)

ModelLevelSuccess ↑Journeys ↑avg Time (s) ↓avg Tokens (k) ↓
LunaSimple6/6 → 6/618/18 → 18/18225.9 → 206.728.06 → 22.34
LunaMedium4/6 → 5/618/33 → 26/33704.5 → 421.843.62 → 28.71
LunaComplex0/3 → 0/30/21 → 0/21timeout → timeout50.14 → 56.90
AstraSimple6/6 → 6/618/18 → 18/18117.2 → 94.78.73 → 8.39
AstraMedium6/6 → 6/632/33 → 33/33585.0 → 301.432.88 → 20.82
AstraComplex3/3 → 3/318/21 → 21/211638.6 → 941.552.91 → 32.68

Q1: Individual Runtimes

The individual runtimes underlying Table 3 are 1694.83 s and 1712.65 s for Default, and 901.48 s and 1082.54 s for Logical Spaces. We observe speedups across different tasks, with the gains depending on the capabilities of the base model, as shown in the table above.

AuthorsOpenReview ↗

A gentle discussion-phase reminder

Dear Reviewer Gj4w,

As the discussion phase draws to a close, we would be grateful for your thoughts on our rebuttal and whether it has addressed the concerns you raised in your review.

Best regards,

The Authors

Comments on the paper

1 comment
AuthorsOpenReview ↗

Summary of Rebuttal and Revisions

Following the author-reviewer discussion period, we are grateful to have received positive and constructive feedback from the reviewers. During the rebuttal phase, the main concerns raised by the reviewers and our corresponding responses and revisions are summarized below:

  1. Consistency Between Generated Videos and World States: We conducted a two-phase quantitative evaluation, demonstrating that our generated videos maintain high geometric and semantic consistency with the given world states.
  2. Robustness of Logical Spaces: We provided expanded ablation studies, verifying that our proposed Logical Spaces remain consistently robust across different task complexities and coding agents.
  3. Expressiveness of the Coding Agent and Diffusion Model: We clarified the necessity and superiority of our diffusion rendering model and included new qualitative visual comparisons in the revised PDF.
  4. Inference Latency: We clarified that our method is fully orthogonal to existing acceleration pipelines and empirically demonstrated that it can be seamlessly adapted to mature few-step distillation methods to achieve real-time performance.
  5. Framework Novelty: We clarified our design as a novel agentic world modeling framework and detailed its distinctions from recent works; following our clarification, Reviewer Gmm2 acknowledged the novelty of our approach, which was one of their initial concerns.

We sincerely thank all the reviewers and the Area Chair for their time, effort, and valuable feedback throughout the review process.

Best regards,
The Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.