- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
CoDeR constructs executable worlds using LLM-based coding agents and employs a video model to synthesize their visual observations. Logical Spaces organize world construction through functional modules and interface contracts. Observation as World Registration projects generated appearance back onto world surfaces for subsequent observations, while Visual Cue Hacking adapts the video model to whitebox inputs. Validation includes WorldScore comparisons, component ablations, a preliminary world-construction experiment, qualitative interaction and evolution examples, and a user study. The results support the system’s feasibility and improved generation performance, although broader claims about persistent dynamic consistency remain less directly evaluated.
Critical strengths
- CoDeR connects automated world construction, executable state transitions, generative rendering, and persistent appearance registration into a functioning system. This enables rules and action consequences to persist independently of the current Traveler’s view, while generated appearance can be reused across observations.
- Table 1 reports the highest average WorldScore among the evaluated methods, and Table 2 shows improvements from registration and Visual Cue Hacking. Appendix A.3 additionally reports reduced construction time under a controlled comparison. Together, these results support the system’s usefulness beyond qualitative demonstrations.
Critical weaknesses
1.The experiments do not sufficiently measure whether generated observations faithfully represent executable state, particularly after objects move, disappear, or change. Visual-quality metrics provide only indirect evidence for persistent identities and event consistency. Appendix A.4 also specifies ordered registration and auxiliary viewpoints for changing regions, which narrow the demonstrated setting and should be reflected more clearly in the main claims. 2. The actual evaluation sample counts, scene and interaction coverage, and handling of failures, retries, and candidate selection are insufficiently specified. Appendix A.3 evaluates only two generations per condition on one construction task; its workflow checks do not establish success rates across independent worlds or diverse interactions. These omissions limit confidence in the broader reliability claims. Missing end-to-end timing, including Artist inference and auxiliary observations, further limits assessment of practical applicability.
Questions for authors
Q1. Tables 1 and 2 report positive quantitative results, but the evaluation coverage and success accounting remain unclear. How many examples were evaluated, which scene and interaction categories were covered, and which videos were scored using the selected VBench metrics? How were reconstruction failures, retries, manual corrections, and candidate selection handled? Beyond the two generations per condition in Appendix A.3, are attempt and success counts available for the demonstrated interaction tasks? Q2. Correct executable state does not automatically guarantee that generated RGB faithfully represents that state. What direct evidence establishes consistency between world state and generated observations, particularly for persistent object identities, event outcomes, and changes during observation gaps? Appendix A.4 specifies ordered registration and auxiliary viewpoints for changing regions. Which reported demonstrations depend on these viewpoints, and how is outdated registered appearance handled when objects move, disappear, or change state? Q3. Appendix A.3 reports a reduction in average world-construction time from 1703.7 to 992.0 seconds, but this does not characterize the complete pipeline. For a representative evaluated setting, what are the separate costs of world construction, Artist inference, and appearance registration or updates? Please specify the hardware, resolution, generated frame count, and whether auxiliary observations are included. This question concerns the practical cost of the current offline framework, rather than requiring real-time operation.
4 comments
W2.2/Q3: End-to-end Latency
Our architecture is fully orthogonal to existing video diffusion acceleration techniques, and the additional runtime overhead introduced by our design is on the order of milliseconds.
The latency introduced by our framework mainly comes from two stages: whitebox world initialization and the Observation as World Registration mechanism.
- Whitebox World Initialization: As reported in the table above, the one-time initialization time is primarily driven by the synthesis of complex cross-system logic; for simple roaming tasks that only require basic scene geometry, initialization takes only 30 seconds to 2 minutes.
- Observation & World Registration Overhead: For Observation (rendering the whitebox and depth videos and encoding them into latents), the latency is near real-time at 56 ms per 22 frames on a single NVIDIA H100 GPU. For World Registration (decoding generated latents back to RGB and registering them onto world surfaces), the measured latency is 61 ms per 22 frames on a single H100 GPU.
- Denoising Generation Latency: Equipped with DMAD (few-step LoRA) under a 4-step inference setting on 32 H100 GPUs, our Artist model generates 124 frames in 6.76 s and 22 frames in 0.92 s, corresponding to ~18 FPS and ~22–24 FPS, respectively. All tests above are conducted at a resolution of 1280 x 704.
Thank you for the clarifications and additional experiments. Most of my main questions have now been addressed. In particular, the evaluation sample counts, expanded world-construction study, and runtime breakdown resolve several reporting gaps identified in my initial review and strengthen the paper’s empirical support.
Thank you for carefully reviewing our responses and additional experiments. We are pleased that most of your main questions have been addressed, and we appreciate your recognition of the strengthened empirical support.
Thank you again for your time and thoughtful assessment.
Dear Reviewer Hz9b,
we sincerely thank you for your recognition of the soundness of our method design and our experimental performance. Below is our detailed response to your constructive comments.
W1/Q2: Consistency between Generated Videos and World States
The generated videos highly match the corresponding world states. To evaluate whether the Artists generate videos that faithfully follow their whitebox inputs, we constructed 30 whitebox worlds spanning indoor and outdoor environments, including bedrooms and cities, as well as more imaginative settings such as underwater environments and alien planets. These worlds feature dynamics ranging from rigid-body to non-rigid motion, including doors opening and closing, vehicles driving, robotic arms manipulating objects, and people walking.
For each world, we sampled a camera trajectory spanning 248 frames and rendered the corresponding whitebox video, producing 30 whitebox videos in total. We then used the Artists to render each video with prompts specifying three different visual styles, yielding 90 videos for evaluation. Our evaluation comprises two phases: (1) alignment between generated videos and whitebox inputs, and (2) retention and updating of registered appearance.
For the first phase, we estimate the depth of the generated videos using Video Depth Anything and evaluate their spatial alignment with the whitebox world using AbsRel and δ₁. The results are reported below. We also visualize examples of relative depth errors in Fig. 15 in the revised pdf.
Depth evaluation alone does not establish the consistency of registered appearance across time, viewpoints, and dynamic conditions. We therefore conduct a second evaluation. We register the previously generated videos into the world, then construct new observation trajectories by retaining the camera positions along the original trajectories while varying the camera orientations. We generate videos along these new trajectories and compute SSIM and PSNR against the original videos within the registered regions.
We evaluate two settings: using the original text prompt and using an empty text prompt, with the registered appearance serving as the sole style cue in the latter setting. The results are reported below. Appearance is registered onto the corresponding 3D mesh surfaces, allowing it to remain associated with objects as they move.
W2.1/Q1: Evaluation Coverage
We have expanded the evaluation across tasks and backbone models, and clarify the sampling, success accounting, and computational costs below. Tables 1 and 2 are evaluated on a stratified subset of 100 WorldScore examples, comprising 60 static examples across all 10 scene categories and 40 dynamic examples across all five motion categories. Each category contains equal numbers of photorealistic and stylized examples. We also extend the Logical Spaces study at three levels of world complexity:
Each world-construction task is independently repeated three times. To further assess robustness across backbone models, we conduct ablations with and without Logical Spaces using GPT-6 Luna with high reasoning effort and GPT-6 Astra with medium reasoning effort. The results are presented below. (Values are reported as Default → w/ Logical Spaces.)