acceptodds
Under review as a conference paper at ICLR 2027

Programmable World Model

21 views

Abstract

Recent video world models generate increasingly realistic and interactive visual experiences, yet lack reliable mechanisms for maintaining persistent world state and enforcing programmable rules over extended interactions. We introduce Programmable World Model, a framework that decouples world-state evolution from visual observation generation. An agent translates natural-language instructions into executable programs that specify entity states and state-transition rules, enabling direct control over individual entities and their interactions. A lightweight engine executes these programs to update and maintain an explicit, persistent global world state, including off-screen entities and non-visual attributes. To connect world state with visual generation, we introduce state-augmented 3D oriented bounding boxes (OBBs) as an intermediate representation. This representation, together with the target camera trajectory, is deterministically compiled into pixel-aligned spatiotemporal conditioning signals for a pretrained video model serving as the generative renderer. This design allows users to create playable games with predefined mechanics, direct control over individual entities, and persistent world state throughout gameplay. We further introduce CombatStateBench, a benchmark for evaluating programmable world models. On CombatStateBench, our method achieves 94% Count Accuracy and 98% State Accuracy, substantially outperforming existing interactive video world models while supporting coherent long-horizon generation. These results demonstrate the effectiveness of separating explicit state evolution from generative rendering for building persistent, programmable worlds.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
World Model

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_BRhZOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

Different from many existing video world models that encode the states of identities and environments in implicit video latents, the paper presents a decoupling strategy to model several state attributes explicitly via coding agent programming, and produce final videos via coarse OBB video prior and video diffusion models. This strategy provides a more controllable, deterministic, and consistent way to maintain the world states and rules. This strategy also helps the video diffusion model to produce more structure-aligned videos, resulting in better long-term spatiotemporal consistency.

Critical strengths

(a) The proposed strategy, which models the state attributes explicitly via coding agent programming and renders final videos via coarse OBB video prior and video diffusion models, is essential for world simulation that requires precise rules, interactions, long-term consistency, and so on. Recent advances of the coding agent demonstrate the potential in image-conditioned 3D object/scene modeling with precise parameter control. The paper also uses this paradigm to strengthen the implicit video world model, leveraging the coding and the video diffusion model to handle the aspects where they have advantages, respectively, i.e., the coding agent is easy to record precise parameters and rules, while the video diffusion model is powerful in rendering high-quality videos.

(b) The paper provides a full-stack pipeline covering a dataset pipeline, a model and training pipeline, and a benchmark. Each part is important in generative model post-training paradigms. The model architecture is also reasonable. The chunk-wise autoregressive long video generation and 3D spatial memory mechanism are widely accepted in the video world model community.

Critical weaknesses

(a) It seems that the quantitative and qualitative results are not enough. And no ablation study is found. For quantitative results, I realize that there are a lot of concurrent works that are tackling similar questions. And comparing to all of them is impossible. However, comparing with some representative models is still necessary, such as HY-World 1.5, Matrix-Game-3.0, LongVie 2, and Astra. For qualitative results, the screenshots in the paper and the videos in the supplementary materials seem relatively simple. For example, the screenshots mainly show a third-person perspective to execute CS-style gunshot behaviors in diverse environments. I realize that the screenshots may come from the visualization of benchmark cases for convenience. However, I think the visualization of more diverse scenario settings is helpful to state "World Model".

(b) The paper is titled "Programmable World Model". However, it seems a bit of over statement, since the demonstrated results are mainly third-person gunshot game videos, and the state attributes are relatively limited. The ability to model a more realistic world is somewhat questionable. I think reducing the scope of the statement would be helpful to clarify the content of the paper. For example, "Programmable Game World Model" is more precise. And I think the scope of "Game World Model" is already enough to be investigated solely in a research paper.

Questions for authors

(a) Real-time interaction is essential in world model concepts. Although early works did not emphasize this point, many later video world models have achieved real-time interaction. The paper uses the chunk-wise autoregressive method to achieve multi-turn long video generation, where the bi-directional denoising is executed in a chunk, and causal denoising is performed across chunks. This paradigm itself is widely accepted in the video world model community. However, it seems that the paper does not discuss acceleration on the proposed model, making the gap between the current inference latency and the real-time interaction unclear. And it is also unclear how quickly the model can respond to input interactions. How long is the temporal latency between user action input and rendered video output?

(b) Long-term generation and consistency are also important in world model concepts. However, current autoregressive generation often suffers from error accumulation, which leads to visual quality degradation. The maximum length of the video results in the supplementary material is about 30 seconds, which is relatively too short compared to many existing minute-level video world models. Besides, while the coding agent can maintain world rules and parameters, it is unclear whether the video diffusion model can render long videos with acceptable error accumulation. I am concerned whether the decoupling strategy can truly generate multi-turn long videos while maintaining consistency.

5 comments
AuthorsOpenReview ↗

Response to Reviewer BRhZ (1/3)

We thank the reviewer for the suggestions on baseline coverage, scenario diversity, scope, interaction latency, and longer-horizon rendering. We address these points separately below. All additions made to the revised manuscript during the rebuttal are marked in blue.

AuthorsOpenReview ↗

Response to Reviewer BRhZ (2/3)

W(a) — Added HY-World 1.5, Matrix-Game 3.0, and Astra as baselines, a racing benchmark, new interaction types and viewpoints, and component ablations.

Additional baselines. We added HY-World 1.5, Matrix-Game 3.0, and Astra to the comparison, alongside LingBot-World v2 and YUME. Table 1 and Figure 3 compare all five baselines with our method on CombatStateBench. Our method ranks first on all eight state-consistency and entity-level control metrics. We did not include LongVie 2 because it addresses a different input setting: it is conditioned on dense depth maps and sparse point maps of the target sequence, which presuppose the geometry of the frames to be generated. Our setting provides only an initial image and programmed OBB trajectories, from which such controls cannot be derived. We discuss LongVie 2 in Related Work.

A racing benchmark. We also compared all methods on a racing benchmark with 27 clips and 11 programmed maneuvers, including braking, spinning, cutting in, and overtaking (Appendix E.1, Table 2, Figure 6). Our method again ranks first on all eight state-consistency and entity-level control metrics. Unlike combat, where states change discretely from alive to dead, racing requires following continuous programmed trajectories and vehicle-to-vehicle interactions. The advantage of our method therefore extends beyond combat-specific state transitions.

Additional scenarios and viewpoints. Following the reviewer's suggestion to broaden scenario and viewpoint coverage, Figure 7 and Appendix E.2 present sitting and standing, object transfer, entering and driving a vehicle, pushing and releasing an object, and first-person traversal. These examples demonstrate the same Programmable World Model framework instantiated with MiniMax-H3 as the generative renderer. A coding agent generates executable world rules, and the engine evolves state-augmented 3D OBBs. The compiler converts these states into a camera-aligned box-control video, supplied together with the initial RGB image and timestamped action descriptions through MiniMax-H3's reference-conditioning interface. These examples provide qualitative evidence of the framework's applicability to additional interactions and viewpoints.

Component ablations. We added two sets of studies in Supplementary Appendix G (Tables 3–4 and Figures 10–11).

Temporal history and spatial memory (Table 3 and Figure 10). These components aim to improve temporal consistency. We therefore measure frame-to-frame visual stability with Temporal stability, and preservation of entity identity at programmed locations over time with Identity at GT location. We evaluate three configurations on ten 30 s clips, using the same initial images, prompts, and programmed controls.

ConfigurationTemporal stability ↑Identity at GT location (%) ↑
Without history (last-frame chaining)98.8561.8
With temporal history99.0268.1
With temporal history + spatial memory99.1578.0

Each component improves both metrics, and combining them performs best. The effect is clearest on Identity at GT location: temporal history raises it from 61.8% to 68.1% (+6.3 points), and spatial memory raises it further to 78.0% (+9.9 points). Temporal stability is already high without history (98.85) and increases steadily to 99.15. Both components therefore mainly help preserve entity identity over long rollouts, with spatial memory contributing the larger gain.

OBB conditioning and state augmentation (Table 4 and Figure 11). This ablation evaluates OBB conditioning and its state augmentation. We choose Entity box IoU to assess spatial alignment, Motion direction to assess adherence to programmed movement, and Re-entry similarity to assess appearance consistency across exit and re-entry. The three variants use the same training duration and the same 14 evaluation scenarios, with shared initial images, prompts, and camera trajectories.

ConfigurationEntity box IoU ↑Motion direction (%) ↑Re-entry similarity ↑
Camera only21.511.9--
3D OBBs without state augmentation57.483.70.644
Ours: state-augmented 3D OBBs62.785.20.751

Our full state-augmented OBB configuration performs best on all three metrics. Re-entry similarity is the cosine similarity between a vehicle's mean DINOv2 appearance features before exit and after re-entry, averaged over eligible events. The higher re-entry similarity (0.751 vs. 0.644) provides evidence that state-augmented 3D OBBs better preserve global identity information across exit and re-entry than geometry-only 3D OBBs, helping entities retain a consistent appearance after being out of view. Detailed metric definitions and evaluation protocols are provided in Section 4.2 and Supplementary Appendices D and G of the revised paper.

AuthorsOpenReview ↗

Response to Reviewer BRhZ (3/3)

W(b) — Clarified the scope: validated mainly in game environments, with broader settings left to future work.

We agree with the reviewer that our validation is conducted mainly in game environments and that the current results do not establish the ability to model more realistic worlds. We will narrow the scope of our claims accordingly.

Within games, the revision broadens the demonstrated settings beyond third-person shooting. The racing benchmark (Table 2 and Figure 6) evaluates continuous vehicle trajectories and vehicle-to-vehicle interactions, and Figure 7 adds posture, object possession, vehicle occupancy, and object contact, as well as a first-person viewpoint.

However, the framework itself is not specific to games. It separates an explicit, executable world state from its generative visual realization, and its state interface supports persistent entities, attributes, relations, and executable rules (Section 3.2 and Figure 2). This design may extend to other domains with structured entities and rules, but we have not validated such settings and leave them to future work.

Q(a) — Action-to-output latency: about 122 ms on one H200.

We verified that the proposed framework can support real-time interaction. To this end, we reduced the chunk size to one latent frame, corresponding to four video frames, distilled a two-step student, and applied engineering optimizations, including CUDA Graphs, fused kernels, FP8 attention and linear layers, a ring-buffer KV cache, and a lightweight VAE. With model inference running on a single H200 GPU at 832 × 480 and a 60 FPS input stream, the system achieves a mean action-to-output latency of about 122 ms.

Of this total, about 50 ms is inherent to processing four-frame chunks at 60 FPS, including waiting for the remaining input frames and presenting the four output frames in order. Model inference accounts for about 36 ms, covering encoding, two denoising steps, and decoding. The remaining 36 ms comes from other overhead, including waiting for the next simulation step, rendering the conditioning frames, and encoding, decoding, and transferring frames.

Q(b) — Added a 60 s rollout with no visible style drift.

We added a 60 s autoregressive rollout, using the same model and generation procedure as the original 30 s tests. Supplementary Appendix F.2 and Figure 9 present generated frames alongside the 3D OBB controls at 1, 15, 30, 45, and 60 s, with forward camera motion and rotation. In this 60 s example, we observe no obvious stylistic drift. This single case does not quantify error accumulation or establish robustness across longer rollouts.

Reviewer_gHosOpenReview ↗
Rating
3
Confidence
4
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

The authors introduce programmable world model, a framework that decouples world-state evolution from visual observation generation. The model keeps an explicit world states with 3D bounding boxes, and a coding agents to convert the language instructions into a program of world. The authors further introduce CombatStateBench, a benchmark for evaluating programmable world models. On the evaluation benchmark, the method achieves 94% count accuracy and 98% state accuracy, substantially outperforming existing interactive video world models.

Critical strengths
  1. The explicit-state design is clean and useful. The authors propose a method the uses a explicit world state to achieve better consistency.

  2. The renderer interface is reasonable and lightweight. Using 3D obbs gives the system a compact geometric representation that can be projected into identity, semantic, and motion controls without requiring full meshes or hand-built animations. This keeps the state programmable while still giving the video model enough spatial guidance.

Critical weaknesses
  1. The benchmark might be too narrow compared to paper's claims. The paper claims that programmable world model achieves much better consistency while the benchmark only consists of 50 combat clips and mainly checks visible character counts and death-state realization. This makes it difficult to tell whether the proposed framework really supports general programmable world behavior, or whether it mainly works well for a narrow set of combat-state transitions.

  2. The state-consistency metrics are too permissive. The metrics of the main benchmark don't fully evaluate instance-level correctness. Count accuracy only checks the total number of visible alive characters. State accuracy passes if the VLM sees at least one dead character after a death event. These metrics do not verify which entity changed state, whether the correct entity stayed dead, whether its location is correct. So a video could still receive a high score even if it violates the state.

Questions for authors
  1. Can authors provide more ways of evaluating programmable rules beyond combat and death events?

  2. How much of the gain comes from explicit state, and how much comes from the stronger spatial conditioning?

4 comments
AuthorsOpenReview ↗

Response to Reviewer gHos (1/2)

We thank the reviewer for the suggestions on benchmark coverage and entity-level evaluation. Following these suggestions, we have expanded the evaluation to additional interaction types and introduced more stringent entity-level measurements. All additions made to the revised manuscript during the rebuttal are marked in blue.

W1, Q1 — Added a racing benchmark and new interaction types beyond combat.

We have expanded the evaluation beyond combat in two complementary ways: a racing benchmark in a different domain and additional interaction types.

Racing benchmark (Appendix E.1, Table 2, and Figure 6). We added a racing benchmark with 27 clips from five initial images and 11 programmed maneuvers, covering braking, spinning, cutting in, and overtaking. Quantitatively, our method ranks first on all state-consistency and entity-level control metrics in Table 2. Vehicle State Program Adherence (SPA-strict) checks whether the designated vehicle follows the programmed trajectory, timing, and persistence while preserving other vehicles. The evaluation protocol is described in Supplementary Appendix E.1, “Quantitative comparison.” Qualitatively, Figure 6 compares all methods on a lead-vehicle spin and an overtaking maneuver.

Additional interaction types (Appendix E.2 and Figure 7). We present qualitative examples of posture changes, object transfer, vehicle occupancy, object contact, and first-person traversal. These examples demonstrate the same Programmable World Model framework instantiated with MiniMax-H3 as the generative renderer. A coding agent generates executable world rules, and the engine evolves state-augmented 3D OBBs. The compiler converts these states into a camera-aligned box-control video, supplied together with the initial RGB image and timestamped action descriptions through MiniMax-H3's reference-conditioning interface. These examples provide qualitative evidence of the framework's applicability to additional interactions and viewpoints.

These additions evaluate programmed vehicle behavior and demonstrate a wider range of state changes and interactions through the same coding-agent and state-augmented-OBB interface. The racing benchmark provides quantitative and qualitative evidence beyond combat, and the additional examples provide qualitative evidence across further interaction types.

AuthorsOpenReview ↗

Response to Reviewer gHos (2/2)

W2 — Added stricter state-consistency metrics.

We added the strict State Program Adherence (SPA-strict) metric to check that the designated entity changes state at the programmed location and within the required time window and maintains the resulting state, with no unintended bystander transitions. Detailed definitions and evaluation protocols are provided in Section 4.2 and Supplementary Appendix D, “SPA-strict for combat”.

Baselines receive the prescribed state-transition information through the prompt at every generation chunk. However, realizing an event from text does not ensure that the designated entity changes state at the required time and location. Such a result can pass the permissive State Accuracy check while failing SPA-strict, which requires all criteria to hold jointly. This distinction helps interpret the zero baseline scores in Table 1. Our per-frame, camera-aligned entity controls provide explicit spatial and temporal guidance, supporting more precise realization of the programmed transitions.

The full comparison in Table 1 shows that our method ranks first on all state-consistency and entity-level control metrics.

Q2 — Added an ablation of spatial conditioning and explicit state.

We quantify the two contributions with an ablation in Supplementary Appendix G, “OBB conditioning and state augmentation” (Table 4 and Figure 11). Three variants add one component at a time: camera-only conditioning, 3D OBBs without state augmentation (adding spatial conditioning), and state-augmented 3D OBBs (additionally exposing explicit entity states, namely instance, category, and direction, to the renderer). All variants use the same training duration and the same 14 evaluation clips, in which people and vehicles enter and leave the camera view, with shared initial images, prompts, and camera trajectories.

We use three metrics. Entity box IoU measures spatial alignment: each person and vehicle is tracked with SAM3, and its tracked box is compared with the programmed box, with missed detections scoring zero. Motion direction measures adherence to programmed movement: it is the fraction of 0.5 s windows in which the tracked and programmed image-space displacements point in the same direction (cosine ≥ 0.5), pooled over 135 eligible windows. Re-entry similarity measures appearance consistency for vehicles that leave and return to the view: it is the cosine similarity between a vehicle's mean DINOv2 features before exit and after return, averaged over eligible events. Camera-only control yields no eligible re-entry events, so its entry is “--”.

ConfigurationEntity box IoU ↑Motion direction (%) ↑Re-entry similarity ↑
Camera only21.511.9--
3D OBBs without state augmentation57.483.70.644
Ours: state-augmented 3D OBBs62.785.20.751

Detailed metric definitions and evaluation protocols are provided in Supplementary Appendices D and G.

Thanks for the reply, I have raised my score.

Thank you for your positive response. We still see the original score on our end. Could you please take another look when you have a chance?

Reviewer_yMvQOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper presents Programmable World Model, a framework that decouples explicit world-state evolution from generative video rendering to enable programmable and persistent interactive worlds. The method employs an agent-driven programming interface to define entity states and executable interaction rules while a lightweight engine maintains a persistent global world state. To bridge structured states and visual generation, the authors adopt state-augmented 3D oriented bounding boxes (3D OBBs), which are compiled into pixel-aligned identity, semantic, and motion controls for a pretrained video generator. The framework also incorporates temporal history and geometry-aligned spatial memory for long-horizon generation. Experiments on CombatStateBench demonstrate improvements in state consistency ad visual quality compared with existing interactive video world models.

Critical strengths
  1. Clear motivation and intuitive framework
  2. Practical integration of structured control and generative rendering.
  3. Experimental results are promising.
Critical weaknesses

1. Insufficient evaluation of spatial consistency and entity-level controllability. Although the central motivation that the paper emphasizes is explicit world states and spatially consistent entity-level control, the quantitative evaluation only relies on global character counts (Count acc), death-state detection (State acc) and visual quality. I think State Accuracy alone cannot verify whether the correct entity undergoes the specified transition at the intended location. These metrics provided limited evidence of precise spatial alignment, identity preservation, and adherence to programmed entity-level interactions. Additional evaluations of such entity localization, spatial relationships, and instance-level state consistency are required.

2. Lack of component-wise ablations. The framework combines several components including state-augmented OBBs, structured spatial controls, temporal history, and geometry-aligned spatial memory. However, the experiments do not sufficiently isolate their individual contributions (especially OBBs). Therefore, it remains unclear how much of the improvements stems from explicit world-state modeling, structured conditioning,, or memory mechanisms. Ablations on such individual components would help clarify the necessity and effectiveness of each design choice.

3. Limited validation of fine-grained interactions and generalizability. The training data and evaluations focus on game-like environments with relatively structured or discrete interaction mechanics. Although the supplementary material provides reasonable motivation for choosing OBBs over higher-dimensional representations (e.g., more detailed, limbs.. , etc), the design choice would benefit from empirical validation to substantiate the claimed tradeoff between controllability, complexity and visual fidelity. So, comparisons with alternative spatial representations (e.g., 2D bounding boxes, 3D bounding boxes without state augmentation, or more expressive 3D representations.. ) would help demonstrate the effectiveness of the proposed OBB-based approach.

Also, while the paper presents qualitative results across diverse scenarios, the quantitative evaluation is primarily limited to combat-oriented interactions. Additional qualitative/quantitative results on existing benchmarks or more diverse interaction scenarios would strengthen the evidence for the proposed representations and its generalizability beyond combat-oriented settings.

4. Limited evaluation of challenging out-of-view dynamics. While the supplementary material presents examples involving dynamic camera rotations and initially out-of-view entities, more challenging scenarios involving prolonged absence and subsequent reappearance scenarios remain insufficiently evaluated. In particular, it would be valuable to examine whether an entity can undergo spatial or state changes while outside the camera's field of view and subsequently reappear with the correct evolved state. Such evaluations would provide more direct evidence of the benefits of decoupling persistent world-state evolution from visual observation generation.

Questions for authors

Q1. How accurately do generated videos reflect the prescribed entity states and spatial relationships?

Q2. How much does each design component contribute to the final performance? Ablations are required.

Q3. Could the authors validate the effectiveness of the OBB representation and its generalizability across other interaction domain ?

Q4. How does the model handle exit-and-reentry scenarios where entities undergo state changes while outside the camera's field of view?

Q5. What happens if the coding agent generates incorrect or inconsistent rules? Can the system detect or recover from such errors, or do they propagate into subsequent world states and generated videos?

6 comments
AuthorsOpenReview ↗

Response to Reviewer yMvQ (1/3)

We thank the reviewer for the detailed comments on entity-level evaluation, representation design, generalization, and persistent state evolution. We group overlapping weaknesses and questions below, retaining their original identifiers and identifying the experimental additions made during the rebuttal. All additions made to the revised manuscript during the rebuttal are marked in blue.

W1, Q1 — Added six entity-level metrics

We added six entity-level metrics in Section 4.2, Table 1, and Supplementary Appendix D. Together with the original Count accuracy and State accuracy, they form eight state-consistency and entity-level control metrics, and our method ranks first on all eight against the five baselines. The excerpt below from Table 1 pairs each property raised by the reviewer with its metric.

Property raised by the reviewerAdded metricOursBest baseline
Entity localizationEntity box IoU ↑63.039.4
Entity localizationCenter error ↓0.0280.055
Spatial relationshipsPairwise left–right order ↑99.995.1
Identity preservationIdentity at GT location ↑77.754.4
Adherence to programmed motionMotion direction ↑89.832.3
Instance-level state adherenceState Program Adherence (SPA-strict) ↑80.00.0

“Best baseline” is the best score among the five baselines for each metric.

  • Entity box IoU measures overlap with the programmed boxes and penalizes missed detections.
  • Center error measures the distance between detected and programmed entity centers.
  • Pairwise left–right order checks whether each pair of entities keeps its programmed left–right order in the image.
  • Identity at GT location checks whether the entity at its programmed location matches the intended appearance reference.
  • Motion direction checks whether each entity moves in its programmed direction.
  • SPA-strict requires the designated entity to change state at the programmed location, within the required time window, and to keep the new state, with no unintended bystander transitions.

The largest gap is on SPA-strict, where all five baselines score 0.0. This is because baselines receive the prescribed transitions only as text at every generation chunk. Text can make the event happen, but it does not pin down which entity changes state, when, or where, so every baseline event misses at least one SPA-strict criterion. In contrast, our compiler supplies per-frame, camera-aligned controls for each entity, which specify all three.

Matching, eligibility, and evaluation protocols are detailed in Supplementary Appendix D (“Localization and identity” and “SPA-strict for combat”).

AuthorsOpenReview ↗

Response to Reviewer yMvQ (2/3)

W2, Q2 — Added ablations of temporal history, spatial memory, and OBB state augmentation

We added two sets of ablation studies in Supplementary Appendix G (Tables 3–4 and Figures 10–11).

Temporal history and spatial memory (Table 3 and Figure 10). These components aim to improve temporal consistency. We therefore measure frame-to-frame visual stability with Temporal stability, and preservation of entity identity at programmed locations over time with Identity at GT location. We evaluate three configurations on ten 30 s clips, using the same initial images, prompts, and programmed controls.

ConfigurationTemporal stability ↑Identity at GT location (%) ↑
Without history (last-frame chaining)98.8561.8
With temporal history99.0268.1
With temporal history + spatial memory99.1578.0

Each component improves both metrics, and combining them performs best. The effect is clearest on Identity at GT location: temporal history raises it from 61.8% to 68.1% (+6.3 points), and spatial memory raises it further to 78.0% (+9.9 points). Temporal stability is already high without history (98.85) and increases steadily to 99.15. Both components therefore mainly help preserve entity identity over long rollouts, with spatial memory contributing the larger gain.

OBB Conditioning and State Augmentation (Table 4 and Figure 11). This ablation evaluates OBB conditioning and its state augmentation. We choose Entity box IoU to assess spatial alignment, Motion direction to assess adherence to programmed movement, and Re-entry similarity to assess appearance consistency across exit and re-entry. The three variants use the same training duration and the same 14 evaluation scenarios, with shared initial images, prompts, and camera trajectories.

ConfigurationEntity box IoU ↑Motion direction (%) ↑Re-entry similarity ↑
Camera only21.511.9--
3D OBBs without state augmentation57.483.70.644
Ours: state-augmented 3D OBBs62.785.20.751

Our full state-augmented OBB configuration performs best on all three metrics. Re-entry similarity is the cosine similarity between a vehicle's mean DINOv2 appearance features before exit and after re-entry, averaged over eligible events. The higher re-entry similarity (0.751 vs. 0.644) provides evidence that state-augmented 3D OBBs better preserve global identity information across exit and re-entry than geometry-only 3D OBBs, helping entities retain a consistent appearance after being out of view. Detailed metric definitions and evaluation protocols are provided in Section 4.2 and Supplementary Appendices D and G of the revised paper.

W3, Q3 — Compared OBBs with camera-only and geometry-only variants; added a racing benchmark and new interaction types.

Ablation of the OBB representation. As explained in Appendix A and Figure 4 of the original submission, OBBs encode position, extent, and orientation in a shared world coordinate system, enabling camera-aligned controls while leaving fine geometry and articulation to the renderer. State augmentation adds persistent identity and semantic and dynamic attributes. The OBB ablation in Appendix G, Table 4, and Figure 11 compares camera-only conditioning, 3D OBBs without state augmentation, and our full controls. As summarized in our response to W2/Q2, the full configuration performs best on all three metrics, providing empirical support for localization, motion control, and appearance consistency across visibility changes.

Cross-domain evaluation. We added a racing benchmark with 27 clips from five initial images and 11 programmed maneuvers, covering braking, spinning, cutting in, and overtaking. Appendix E.1, Table 2, and Figure 6 are new rebuttal additions. Our method outperforms all five baselines on every state-consistency and entity-level control metric, extending the quantitative evidence beyond combat.

Broader interactions. Appendix E.2 and Figure 7 apply the same Programmable World Model framework beyond combat, covering sitting and standing, object transfer between people, entering and driving a vehicle, pushing and releasing an object, and first-person traversal. A coding agent generates executable world rules, and the engine evolves state-augmented 3D OBBs. The compiler converts these states into a camera-aligned box-control video, supplied together with the initial RGB image and timestamped action descriptions through MiniMax-H3's reference-conditioning interface. These examples provide qualitative evidence that the framework applies to interactions and viewpoints beyond combat.

AuthorsOpenReview ↗

Response to Reviewer yMvQ (3/3)

W4, Q4 — Added an exit-and-reentry test: states changed off-screen are rendered correctly on return.

An entity’s visibility does not determine whether its canonical state is maintained or updated. As described in Sections 3.2–3.3 of the original submission, the engine advances entity states according to the world program, and the compiler projects the updated state under the current camera. When an entity re-enters the view, the rendering controls therefore reflect its current state.

We added an experiment specifically targeting this setting in Appendix F.1 and Figure 8.

In this 30-second sequence, two entities are initially alive and upright, then remain outside the camera view for approximately 10 seconds. Their programmed states change to dead at 12 seconds, while they are out of view. When the camera returns, both are rendered down at 19 seconds and remain down through the 30 s endpoint.

This example directly tests a state transition that occurs during an entity's absence and shows that the evolved state is rendered after re-entry and maintained through the end of the sequence.

Q5 — Clarification on how incorrect rules are detected and when they can propagate.

We check generated world programs through engine-level validation and unit tests. Runtime checks cover entity creation, structured commands, collision and movement constraints, combat and death, vehicle occupancy, and resetting the world. Invalid entity specifications and commands are rejected, while supported engine operations enforce the corresponding world rules. The unit-test suite covers scene schemas, movement, reset behavior, combat and death, perception states, agent independence and reproducibility, runtime commands, seeded randomness, and semantic logs. When validation fails, the generated world program is revised and re-executed, and the validation checks are rerun before proceeding.

However, these checks do not guarantee that every rule matches the intended instruction. An incorrect rule can generate internally consistent states and annotations; if it passes the checks, the error can propagate into subsequent world states and generated videos.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.