acceptodds
Withdrawn from ICLR 2027

State World Models: Execute the Rules, Generate the Pixels

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

39 views

Abstract

Existing world models can generate action-conditioned video from a single image. Many focus on camera movement, allowing users to explore the generated scene. However, navigation alone does not ensure that an action follows persistent rules when its outcome depends on state that is invisible in the current frame. We present **Stateful Harness**, a framework that separates world evolution from video rendering to support stateful and rule-governed interaction. It maintains an evolving world state under specified rules, renders each resulting event with a frozen video model, and uses the generated clip to update the world state for the next turn. At each turn, the harness resolves the action selected by the user and compiles the resulting event into an instruction, while a preview clip from an editable 3D scene provides spatial and camera guidance. To evaluate rule compliance, we build **WorldState-Bench**, a benchmark of 50 counterfactual twin pairs that share an initial frame and action but differ in hidden state and required outcome. Under rule-only prompting, the best of the 11 evaluated video backends produces the required endings for both cases in 32% of pairs. For MiniMax-H3, specifying the required endings raises this rate from 32% to 64%, while the full harness reaches 92% with the same frozen model. These results support separating rule execution from video rendering for world interactions with hidden state.

Primary area:
generative models
Keywords:
Diffusion, World Models, Interactive Video Generation, Executable Rules, Counterfactual Benchmark

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_vnJ4OpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

The paper present a video game benchmark. It assumes some sort of state machine or an internal representation. Then using that information a vlm uses a genAI model to generate the interaction and evaluate the outputs. It makes sure that the agent

Critical strengths

There is benchmark introduced.

Critical weaknesses

The paper in its current format is too hard to read, the complexity of the sentences are too high. I am sure the content is interesting, but if I cannot read this paper as a native speaker, I doubt others can actually learn much from it.

Questions for authors

Here are my raw notes while reading the abstract / intro.

bstract. No need to speak about camera control. “support stateful and rule-governed interaction” this is a little vague, what does these words mean. “It maintains an evolving world state under specified rules, renders each resulting event with a frozen video model, and uses the generated clip to update the world state for the next turn” this sentence is overwhelming. event? what is the world state. “at each turn”? which turn? “instruction”? like prompt? “3d scene”? How is this hidden state constructed? like the benchmark is for a single object.

I dont understand why this paper is motivated by the navigation world models. I think controlling rendering artefacts is strong enough. I think the intro is clumsy, it does not clearly state the problem early. Sounds like control based on hidden state is the problem. Abstract failed at stating that. “Given a world description and an initial image” are the authors describing a game world? This is confusing. “select an actions” from where? Is the 3d scene an actual game state? What is described in the text of the intro is far more complex than the figure 1. “Together, rule execution, video generation, and visual observation form a closed loop that yields state world models.” So it is a game loop that is slow? Again “hidden state” what is this state? How is the state connected to rendering? Change-to-Comply protocol, what is that? this intro is confusing. looking at table 1, why is there a benchmark if the authors already solved it?

2 comments

Thank you for the questions. We will improve the abstract and introduction.

  1. Problem and motivation. We study whether generated videos follow explicit state and rules. For example, opening a chest should succeed only when the player has a key, even if the images are identical. Navigation is background context as it is the current research focus of world models. We think only navigating the world is not enough if not modeling how it should evolve.
  2. State and actions. A coding agent converts a game-world description into state variables, available actions, and executable rules (§3.1). Hidden state is an explicit variable invisible in the image, such as key ownership. Its value is initialized by the program and supplied to the system. “Stateful” means state persists across turns, and “rule-governed” means rules determine permitted changes. Users select actions defined by the program.
  3. Generation and interaction. A turn executes one action, determines an event, generates its video, and updates state. An “event” is the resolved outcome, and an “instruction” is its compiled text prompt. The image-derived 3D scene supplies optional layout/camera guidance and represents only spatial state. Its current implementation uses a single subject. VLM observations can update state only when compatible with the rules (§3.2). Our current design is a turn-based loop, with no real-time performance claim.
  4. Why the benchmark is necessary. We do not claim that the problem is solved. WorldState-Bench evaluates how well existing models and our framework address it and identifies remaining failures. With the same frozen MiniMax-H3 backbone, PairFollow improves from 32% under Rule-only and 64% under Rule+end to 92% with our harness, and PairStrict improves from 4% under Rule+end to 90%. These results demonstrate progress while leaving room for improvement.

We acknowledge the presentation concerns. We would also welcome the opportunity to address any methodological, experimental, or soundness concerns you may have.

Thank you for the great answer. But to me this paper is not ready for publication and I will not change my score.

Reviewer_R6xFOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

The paper introduces a training-free framework named Stateful Harness. This framework uses a coding agent to record and update the world state. The output results of the coding agent are transformed into text prompts, image and camera references, and coarse spatial proxy video priors to formulate a set of inputs to off-the-shelf video generators. The paper mainly emphasizes the design of the coding agent process, leaving the video generation model frozen. This framework can generalize to diverse game scenarios, since the coding agent can plan different world rules given user requests, and the video diffusion model can render diverse visual results. The paper also proposes a benchmark to examine whether Stateful Harness and other baseline models can correctly respond to different hidden states when given the same observations.

Critical strengths

(a) The paper implements several up-to-date techniques, such as coding agent harness, coarse spatial proxy video priors, and a state-of-the-art open-source video generator (MiniMax-H3). The idea that a coding agent tracks and updates world states while a video generator renders visual observations is reasonable and can take advantage of both sides.

Critical weaknesses

(a) It seems that the states and rules recorded in the coding agent cannot preserve the world effectively. For example, in Figure 15 of the paper and "Exploration.mp4" of the supplementary materials, the visual identity of the character changes largely from "act 1" to "act 3". Additionally, the scene from "act 8" to "act 10" also suffers an obvious drift. Due to this issue, I am questionable that, for example, whether the blue banner in "act 8", which interacts with users, still exists and keeps its state. This may partially owing to the frozen video generation model, which itself cannot preserve long-term consistency. However, since the proposed framework is designed to track world evolution, it seems to fail to make a remarkable mitigation in long-term visual inconsistency.

(b) The inference latency remains unclear, which is important in world model applications. The harness process contains several stages before constructing the input for the video diffusion model. The update of parameters may be fast. However, the construction of "Visual Script" and "Prompt Compiler" still requires time. And after the video diffusion model performs multi-step denoising, the framework needs a vision-language model to perform verification. This stage also introduces latency.

Questions for authors

(a) See Critical Weaknesses (a). It seems that the states and rules recorded in the coding agent and the corresponding spatial proxy video priors cannot preserve long-term visual consistency. I wonder whether there are some additional mechanisms helping to maintain the consistency that I may miss.

(b) See Critical Weaknesses (b). I wonder the temporal latency from a player's decision to the final output video.

1 comment

Thank you for pointing out the visual drift and asking about latency.

 

Q1. Why does visual drift occur, and what does Stateful Harness preserve?

The visual drift in Figure 15 mainly reflects the video model’s limited ability to keep appearances consistent across clips. We use MiniMax-H3 with frozen weights and no additional training for cross-turn consistency. It generates a separate clip each turn. Our harness provides the previous clip’s final frame and the spatial preview as reference cues for appearance, layout, and camera motion, but they offer limited control over visual consistency (how characters and objects look) in each new clip.

The spatial preview improves consistency. In the WBench navigation split (Table 9), we kept the prompt template fixed and added the preview. The trajectory score rose from 69.0 to 77.7, the consistency score from 75.0 to 79.2, and the overall navigation score from 76.8 to 79.0. However, maintaining an exactly consistent appearance over many turns remains a limitation. Additional references for characters, objects, and scenes are one possible extension.

The main contribution of our harness is to maintain and update the world state according to the rules across turns. Improved spatial consistency is a secondary benefit. Stateful Harness records values such as stamina and quest progress, applies the rules to resolve each action, and carries the updated state into the next turn. Figures 15–17 show how the rules use this state in the three demos.

In Act 8 of Figure 15, the harness records the player’s interaction with the blue banner in the textual game state. The banner’s appearance may vary across clips, while the recorded blue banner state persists across turns. The rule engine uses the stored state to resolve later actions, so a discrepancy in the banner’s rendering alone does not change how the quest progresses.

 

DemoState across turnsHow the rules use or update the state
ExplorationStamina, exhaustion status, quest progressWhen stamina reaches zero, the player can only walk slowly. Rest brings stamina back to 60 in the demo. Quest progress carries across scene changes and advances from 1/3 to 2/3 to 3/3.
DuelPlayer and opponent health (HP), player manaThe game checks whether the player has enough mana to use a skill. Damage reduces HP, and the game declares victory when the opponent’s HP reaches zero on turn 11.
RelationshipAffection score, relationship stageAs affection rises from 47 to 75, the characters go from friends to flirting on turn 14 and become a couple on turn 29. The resulting relationship state allows the final confession to succeed.

 

Q2. What is the latency from a player’s decision to the output video?

We profiled single-turn WorldState-Bench and three multi-turn demos. MiniMax-H3 rendered all clips at 864×480 on 4 A800 GPUs. The column headers give the duration of the generated clips. The entries report mean wall-clock time in seconds per clip or turn.

StageWorldState-Bench (5 s)Duel  (10 s)Exploration (5 s)Relationship (5 s)
Propose available actions—0.000510.61 (includes VLM)—
Resolve the turn (Rule Engine)0.00010.001013.57 (includes VLM)4.87 (includes LLM)
Build VisualScript0.00010.0003<0.00010.0001
Compile prompt0.00010.00020.00020.0002
Render video11.39–11.7126.9111.9611.48
Sample frames after rendering——0.95—
Reported total11.39–11.7126.9237.0916.35

In WorldState-Bench (5 s clips) and Duel (10 s clips), the harness executes fixed rules directly in code. Turn resolution averages 0.1 ms and 1.0 ms, respectively. Building the VisualScript and compiling the prompt each average at most 0.3 ms in these settings, so video rendering dominates the measured latency.

We use a frozen Qwen3.8-27B to support player interaction in the exploration and relationship demos. In exploration, the model uses three frames sampled from the previous clip to propose actions grounded in the visible scene and plan how the selected action should unfold on screen. In the relationship demo, the same model generates the character’s reply to the player. The harness uses these outputs to guide interaction and rendering, while the rule engine updates the game state.

The current inference pipeline has not been optimized for latency. Using smaller, task-specific models for action proposal and scene planning could reduce inference time. Streamlining prompts and running independent operations in parallel could further reduce per-turn latency.

Reviewer_3e7wOpenReview ↗
Rating
3
Confidence
4
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

The paper separates rule execution from video rendering. Stateful Harness resolves each action with explicit state and rules, then renders the settled event with a frozen video model. On WorldState-Bench, the method substantially outperforms the other approaches.

Critical strengths

The problem is well motivated and interesting. Figure 2 states the research question and the motivation directly. The experiments are thorough and evaluate a large number of models.

Critical weaknesses

In the cases shown in Figures 7 and 8, the State-Harness output uses the same format as MiniMax-H3’s native prompt format, h3-context-ir. This format and distribution are those that best elicit the base H3 video model’s generation ability, whereas the baseline inputs appear comparatively unstructured. Is most of the reported gain attributable to the input prompt format for the specific model?

What is the time overhead of State-Harness relative to generating the video directly, and what is that overhead in the multi-turn setting?

Questions for authors

The multi-turn cases appear to be shown only in Section 5.3. For the State-Harness runs underlying the main WorldState-Bench results in Table 1, what is the actual distribution of the number of turns executed?

1 comment

Thank you for your questions about the prompt comparison, latency, and evaluation scope.

 

Q1. Does MiniMax-H3’s native prompt format explain the gains?

We agree that prompt format was a potential confound in the original comparison. We initially expected the template alone to have little effect because it adds no information about how the current state affects the outcome. To verify this, we matched the prompt templates of both MiniMax-H3 baselines to the template used by our Stateful Harness and reran WorldState-Bench with the same evaluation protocol.

We report the six metrics (%) from Table 1 and Table 4 of the paper.

ConditionENDFOLLOW↑PAIRFOLLOW↑TWINCONTRAST↑STRICT↑CNTABSENT↑PAIRSTRICT↑
Rule-only64303226300
Rule-only + harness template65303426304
Rule+end79585829304
Rule+end + harness template74505428320
Stateful Harness (paper)969296959890

Matching the template did not consistently improve either baseline. Stateful Harness still followed the rules more reliably than both baselines. These results support our claim that executing rules on explicit state and using that state to guide rendering gives us better control over the generated video.

 

Q2. What is the overhead in single- and multi-turn settings?

We profiled single-turn WorldState-Bench and three multi-turn demos. MiniMax-H3 rendered all clips at 864×480 on 4 A800 GPUs. The column headers give the duration of the generated clips. The entries report mean wall-clock time in seconds per clip or turn.

StageWorldState-Bench (5 s)Duel (10 s)Exploration (5 s)Relationship (5 s)
Propose available actions—0.000510.61 (includes VLM)—
Resolve the turn (Rule Engine)0.00010.001013.57 (includes VLM)4.87 (includes LLM)
Build VisualScript0.00010.0003<0.00010.0001
Compile prompt0.00010.00020.00020.0002
Render video11.39–11.7126.9111.9611.48
Sample frames after rendering——0.95—
Reported total11.39–11.7126.9237.0916.35

In WorldState-Bench (5 s clips) and Duel (10 s clips), the harness executes fixed rules directly in code. Turn resolution averages 0.1 ms and 1.0 ms, respectively. Building the VisualScript and compiling the prompt each average at most 0.3 ms in these settings, so video rendering dominates the measured latency.

We use a frozen Qwen3.8-27B to support player interaction in the exploration and relationship demos. In exploration, the model uses three frames sampled from the previous clip to propose actions grounded in the visible scene and plan how the selected action should unfold on screen. In the relationship demo, the same model generates the character’s reply to the player. The harness uses these outputs to guide interaction and rendering, while the rule engine updates the game state.

The current inference pipeline has not been optimized for latency. Using smaller, task-specific models for action proposal and scene planning could reduce inference time. Streamlining prompts and running independent operations in parallel could further reduce per-turn latency.

 

Q3. How many turns were executed for Table 1?

All 100 WorldState-Bench instances in Table 1 run for one turn each. In each of the 50 pairs, the starting image and action are the same, but the hidden game state differs. The benchmark tests whether our harness can apply the game rules to the current state, resolve the action, and render the result.

Most baselines struggled to render the correct state-dependent outcome on the first turn (Table 1). Given these first-turn failures, we limited the quantitative WorldState-Bench evaluation to single-turn state transitions. We evaluated consistency and other aspects of multi-turn video generation separately on WBench (§5.4, Tables 2 and 7–9).

The three multi-turn demos cover exploration (15 turns), a relationship narrative (29 turns), and a duel (11 turns). They demonstrate how our harness maintains game state across turns through successive rule-governed updates (§5.3, Figures 15–17).

Comments on the paper

1 comment

Dear Reviewer,

Thank you again for your review. We wanted to briefly follow up on our rebuttal. If you have a chance, we’d appreciate hearing whether our response addresses your concerns or if there are any points we could clarify further.

Thank you for your time and consideration.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.