acceptodds
Under review as a conference paper at ICLR 2027

Beyond Pixels: Grounding Video World Models with State-Centric Information

Abstract

Video world models are trained under the paradigm of predicting future video frames from a stream of input actions, with the hope of unlocking fast, interactive and physics-accurate simulation environments from video data only. Many state-of-the-art world models rely on off-the-shelf VAE encoders of text-to-video generative models, thus inheriting from their ad-hoc latent space, which is also hard to interpret or control, and is not built with the task of world modelling in mind. In contrast, in settings where hard-coded physics simulations are standard (e.g. autonomous driving or robotics), world models often rely on explicit 3D, sensor information or dedicated logic engines. In this work, we study how privileged state information can be used alongside video data to unlock new properties of video world models: We first show that a modular training recipe enables improved efficiency, while modeling both the states and video streams jointly induces a natural measure of uncertainty of the full world state, which may only be partially observable in the video. We also exploit the structure of the state space to infuse LLM priors during training as a form of data augmentation to correct policy bias or introduce novel actions. Finally, we design state-based metrics for the Rocket Science dataset to evaluate the world model's physics understanding.

Primary area:
generative models
Keywords:
world models, controllable, interpretability

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_HvidOpenReview ↗
Rating
2
Confidence
2
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper investigates how explicit physical state information can improve video world models. The authors explore two architectures: a State-Centric World Model (SCWM), which decouples physical state transitions from video rendering, and a Dual-Stream World Model (DSWM), which jointly models video and physical states through cross-attention. Additionally, the authors introduce RoLeSynth, an LLM-assisted synthetic data generation approach to mitigate policy bias and improve behavioral diversity. Experiments on the Rocket Science dataset demonstrate that SCWM achieves better modeling efficiency and video generation quality, while DSWM enables uncertainty estimation of partially observed states. The authors also validate the effectiveness of synthetic data augmentation in reducing policy bias and improving generalization across different behavior distributions.

Critical strengths
  1. Strong empirical results and efficiency improvements. The proposed SCWM achieves competitive video generation quality while significantly reducing the computational cost of dynamics modeling. The experiments demonstrate that a small state transition model can achieve performance comparable to much larger video world models, highlighting the efficiency benefits of explicit state modeling.

  2. Systematic exploration of state-based world modeling. The paper investigates two different architectures, SCWM and DSWM, and provides detailed experiments and ablation studies to analyze their performance and trade-offs. The DSWM also demonstrates interesting capabilities in estimating hidden physical states and their uncertainty.

  3. Interesting analysis of policy bias and data augmentation. The authors identify policy bias in the Rocket Science dataset and introduce RoLeSynth to generate diverse synthetic state trajectories. The experiments provide useful insights into the distinction between learning physical dynamics and learning behavior distributions, and demonstrate the benefits of synthetic data augmentation for generalization.

Critical weaknesses
  1. Unclear research motivation and methodological contributions. The paper investigates several directions, including state-centric world modeling, dual-stream generation, uncertainty estimation, and LLM-assisted data augmentation. However, these directions appear relatively independent, and similar ideas have already been explored in previous works. It remains unclear what specific research problem the paper primarily aims to address and what new methodological contributions or generalizable insights it provides beyond existing approaches. A clearer motivation and a more focused discussion of the key contributions would strengthen the paper.

  2. Insufficient comparisons with existing state-aware world models. The main experiments primarily compare the three architectures implemented by the authors, while closely related approaches such as Agora-1 and StatePlay are not directly evaluated. Given the similarities between these methods and the proposed SCWM and DSWM, more direct comparisons or controlled architectural ablations would help establish the independent contributions of this work.

  3. Potential confounding factors in the experimental evaluation. The SCWM transition model is trained with approximately nine times more state-only data, and SCWM relies on explicit initial state information, whereas VWM primarily uses visual observations. These differences make it difficult to determine whether the reported improvements come from the proposed architecture, additional training data, or privileged state information. More controlled experiments under comparable training and input settings would strengthen the conclusions.

Questions for authors

N/A

3 comments
AuthorsOpenReview ↗

Response to reviewer Hvid

Unclear research motivation and methodological contributions

Our goal is not to introduce a new architecture for world modelling but rather present a systematic study of integrating state-centric information in the standard video world models paradigm in a controlled setting. As such, even though SCWM and DSWM do share similarities with previous technical contributions, we believe we provide the first in-depth analysis of these architectures with equal training procedure. We also demonstrate additional features enabled by state-centric models which were not highlighted in previous work to the best of our knowledge: policy bias correction (Section 4.2), controllability via state manipulation (Appendix F1), uncertainty estimation (Section 4.3) and mixed-supervision training (Appendix F3). We have updated the introduction accordingly.

Potential confounding factors in the experimental evaluation

Data Size. We would like to clarify that the SCWM is trained on nine times more steps but not more data: The transition model of the SCWM train on more epochs of the same dataset. This is possible because the transition model is efficient to train (45 GPU hours vs 423 for the VWM), which we see as a strength of SCWM. We also evaluate the SCWM at earlier training stage of the transition model to match the number of samples seen in training (Table C). Overall, training the VWM for longer does not improve its visual quality. The same is true for SCW but the state prediction does improve with more training.

Table C. Matched data budgets.

ModelClip samplesgFID ↓gFVD ↓DINO drift ↓gPSNR ↑Pos. RMSE ↓Ball pos. err. ↓
SCWM at the VWM's budget12M9.4984.30.22115.1820943014
SCWM (paper)109M9.6186.50.21515.3619572808
VWM (paper)12M10.37103.20.22015.12––
VWM trained twice as long24M10.48104.10.22115.10––

States as prefix. Thanks, this is an interesting question.

  • (i) For the VWM and SCWM, both receives an 4s identical ground-truth prefix but the modality of it (state vs frames) is dictated by the architecture. In Table D, we estimate (a) the SCWM with states prefix estimated from frames (rather than ground-truth) and (b) the VWM with prefix rendered from ground-truth states (Table E, top rows) . The results show the VWM does not benefit from knowing ground-truth states, while SCWM does exploit this information.
  • (ii) For the DSWM however, it can by design receive as prefix either (a) states, (b) frames or (c) both (paper setting) which allows for a fairer comparison. As shown in Table E (bottom rows) including states in the prefix generally leads to better visual quality (both for the SCWM and DSWM) which shows state-centric information is also well exploited by DSWM.

Table D. Changing the modality of the 4 s ground-truth prefix.

ModelPrefixgFIDgFVD
SCWMGT states (paper setting)9.6286.6
SCWMStates estimated from the ground-truth frames by the DSWM11.09109.3
VWMGT frames (paper setting)10.60104.7
VWMFrames rendered from the ground-truth states by the SCWM renderer10.51105.9
DSWMGT states and frames (paper setting)10.8996.9
DSWMGT frames only11.70104.9
DSWMGT states only11.20103.7

Link to Agora-1 and StatePlay

We have extended our citations of both work in the updated manuscript to provide additional context. We give a short summary comparison below and add a more detailed commentary in the global response above.

  • Agora-1 [18 May 2026] falls into the category of SCWM; However, it was only released as a blog post with little to no technical details on the architecture or results, which is why we do not use it as a baseline.
  • StatePlay [29 July 2026] is a finetune of WAN2.2-5B model with an additional state stream next to the pixel diffusion backbone, making it similar to DSWM. a core difference is that the noise level of the state stream and frame stream is equal in StatePlay: In contrast, our DSWM samples  different noise levels for states and frames (Section 3.2), which unlocks two additional inference modes: Inverse renderer (clean video in, denoise state, see Section 4.3) and Frame renderer (state in, denoise frame). For the same reason, StatePlay can only be trained with paired state-frame data, while DSWM can be trained with mixed supervision (Appendix F3).

I appreciate the authors' detailed rebuttal and additional experiments, particularly the controlled comparisons of training budgets and initial state conditions. These results help clarify several of my experimental concerns.

However, my primary concern remains the limited methodological novelty and generalizable insights. While the authors position the paper as a systematic study rather than a novel architectural contribution, I believe such a study should establish meaningful and broadly applicable conclusions beyond a single experimental environment.

In particular, the additional experiments show that SCWM's advantages depend substantially on access to accurate ground-truth states, with its video generation performance deteriorating when states are estimated from visual observations. This raises an important question about the conditions under which state-centric modeling is actually preferable to video-based modeling.

Furthermore, although the authors clarify the differences from existing state-aware world models, the independent contributions over prior approaches remain insufficiently established.

Overall, I acknowledge the empirical value of the work, but I do not find that the current study provides sufficiently novel or generalizable insights to justify acceptance at ICLR. I will therefore maintain my original rating.

Thank you for your response and acknowledgement of the rebuttal; We would like to address three short points below.

“Furthermore, although the authors clarify the differences from existing state-aware world models, the independent contributions over prior approaches remain insufficiently established."

Beyond the systematic study, we have restated additional contributions in the general response (policy bias correction (Section 4.2), controllability via state manipulation (Appendix F1), uncertainty estimation (Section 4.3) and mixed-supervision training (Appendix F3)) and differences with recent related work (Agora-1, and mixed supervision setting DSWM in Appendix F3 as compared to StatePlay). We would be grateful if the reviewer could point to the specific contributions they find insufficiently supported, so that we can address them directly.

“This raises an important question about the conditions under which state-centric modeling is actually preferable to video-based modeling."

We respectfully disagree that this is a weakness of our work specifically: this is a criticism of state-aware models in general. Moreover, there is a growing body of work showing benefits of state-aware modelling, for instance for controllability or efficiency in planning/simulation applications. Here is a quick selection of recent work on games and robotics:

  • Ye et al. PointCast: One World Model for Rigid, Articulated, and Deformable Object Manipulation. Sep 2026.
  • Deng et al. WorldMind: Decoupled Game World Model for State-Aware NPC Behavior. Aug 2026.
  • Meng et al. Marionette: Predicting World States, Rendering Geometry, Painting Appearance. Aug 2026.
  • Cai et al. MASS: Multiplayer World Models with Authoritative Shared State. Aug 2026.
  • Zhang et al. World Action Planner: Generalizable Robot Decision-Making with Action-Conditioned World Models. Jul 2026.
  • Kim et al. Robot-Factored World Models via Robot Rendering. CoRL 2026
  • Li et al. From Pixels to States: Rethinking Interactive World Models as Game Engines. Jul 2026.

“SCWM's advantages depend substantially on access to accurate ground-truth states…”

We agree, but this result illustrates the core motivation of SCWM: they can leverage states to further improve model performance, which VWM can not by design. Judging SCWM on estimated, rather than ground truth, states is out of distribution as it was only trained on actual states, thus cannot compensate for any error of the inverse renderer. In addition, the required ground-truth state prompt is not necessarily a difficult-to-achieve requirement (e.g. see below a quick study where we measure both models at different prompt lengths, showing we need as little as 4 frames, without severe degradation to visual quality).

Context givenVWM (video) gFID ↓SCWM (states) gFID ↓
4 s10.379.62
2 s10.369.52
1 s10.549.63
0.4 s10.539.82
0.2 s (4 frames)10.429.73
Reviewer_DHQWOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The authors proposed two types of world models to incorporate the states into the video world models: (1) state-centric world model (SCWM) that decouples the rendering and dynamics modeling, (2) Dual-Stream World Model (DSWM) that jointly predicts states and video through two interacting streams.

Critical strengths
  1. The model design is well motivated: It's a sensible choice to decouple the dynamics and rendering, which enables inexpensive state rollouts, and direct inspection.
  2. The improvements, especially the SCWM in Table 1, are promising and consistent across various metrics.
Critical weaknesses
  1. Evaluation: the paper claims to support physics-accurate simulation, but most metrics are about video quality and trajectory errors, not informative physical metrics.
  2. There are two proposed models, SCWM and DSWM, which one do the authors would like to deliver given that they cannot be merged into one? What would be the practical use case?
Questions for authors
  1. Can the authors explain why there is basically no latency differences under different model sizes?
2 comments
AuthorsOpenReview ↗

Response to reviewer DHQW

Physics-accurate metrics

We agree and have rephrased with the more specific term "game physics". In our setting we use the term game physics to differentiate from a game state that would only contain global game information such as the current score or clock. In particular SCWM and DSWM explicitly model the cars/balls positions which we then can use to obtain more interpretable metrics on the states in addition to classical metrics such as FID (e.g in Table 1,2 and 3).

SCWM or DSWM as a flagship model

We believe SCWM and DSWM serve different use cases, but DSWM is the more well-rounded approach:

  • SCWM is interesting in scenarios where (i) decoding to pixel space is not crucial [a,b,c,d]], as the explicit state space structure allows to efficiently train small and lightweight transition models, or (ii) when a rendering engine already exists (e.g. 3D simulation engine) or can easily be trained. The main caveat is it can only be trained with state annotation, which is not always easy to obtain.
  • DSWM  is a more general design and achieves "the best of both worlds": It achieves similar visual quality as the VWM while also being able to leverage and produce state-centric information like SCWM which in particular enables more interpretable metrics (e.g., controlling the position of objects). In addition, DSWM unlocks two additional inference modes in a single model: Inverse renderer (clean video in, denoise state, see Section 4.3 about uncertainty prediction) and Frame renderer (state in, denoise frame). It can also be trained with mixed supervision (Appendix F3), partially lessening the need for video-state paired data. However it is costlier than other models as it needs to handle both streams of information.

Flat latency

Thank you for this question, which led us to re-check and expand our latency measurements. We have modified the corresponding paragraph and added a new figure in the updated manuscript to include more detailed efficiency analysis in the new Figure 4. We report a short summary below.

What is being measured and Errata on Figure 3. In the latency plot of the previous Figure 3 we are measuring the latency of dynamics modelling, at batch size 1, between the SCWM and VWM as this is the main efficiency driver for many applications of world models (e.g. planning or RL training [a, b, c, d]). During analysis we also found one mistake in Figure 3: The SCWM curve measures the transition model alone, but the VWM curve also included the video decoder. We re-measured the VWM, so that both curves now measure the same one step of the dynamics model. In the original setting of the figure (batch size 1, no compilation), one step takes about 70 ms for the SCWM transition model and about 140 ms for the VWM latent dynamics model, thus the SCWM transition model is about 2× faster, instead of about 4×. We have updated Figure 3 accordingly. This does not change the flatness of the curve, which we explain below.

Flat latency. When measuring theoretical FLOPs, we do observe the expected latency decrease with smaller model size. However when measuring walltime, at batch size 1, each step runs many small GPU kernels and most of the time goes into launching these operations, not into computing them (the GPU is idle for much of the step). This overhead causes the apparent flatness of the curve. If we measure the time spent on GPU only with a profiler, then we observe the downwards trend (see updated Figure 4 in the new manuscript).

Batch size. Another way to recover the downwards trend is by increasing batch size, thus increasing the cost of each operation. With a batch size of 64 the SCWM transition model is still flat because it is still not enough work for it: it needs 0.3 to 22 GFLOPs per player and step, against 250 to 3,747 for the VWM world model (see updated Figure 4 in the new manuscript).

[a] MuZero: Mastering Atari, Go and Chess by Planning with a learned model (Schrittwieser et al., Nature 2020) [b] DreamerV3: Mastering Diverse Domains through World Models (Hafner et al., 2023) [c] TD-MPC2: Scalable Robust World Models for Continuous Control (Hansen et al., ICLR 2024) [d] DINO-WM: World Models on Pretrained Visual Features enable Zero-shot Planning (Zhou et al., ICML 2025)

Please find below additional latency measurements to support our previous comment; Or, equivalently please refer to the newly added Figure 4 in the updated manuscript

Latency Table. Runtime for one step (0.1 s of game) of the dynamics model alone in SCWM and VWM (no renderer or decoder) on one H100 GPU (bf16). For the model sizes which do not appear in the paper: We evaluate the latency of random weights checkpoints.

ModelParamsGFLOPs per player and stepBatch size 1, no compilation (ms per step)Batch size 1, compiled (ms per step)64 players in parallel, compiled (ms per step, per player)
SCWM transition model13M0.369.123.70.45
SCWM transition model27M0.670.724.00.46
SCWM transition model51M1.169.623.90.45
SCWM transition model96M2.171.225.00.47
SCWM transition model (paper)197M4.370.925.10.48
SCWM transition model409M9.070.825.40.48
SCWM transition model787M17.271.425.20.48
SCWM transition model983M21.588.226.50.47
VWM dynamics model78M250136.636.82.98
VWM dynamics model145M462145.739.44.61
VWM dynamics model298M945138.336.46.33
VWM dynamics model619M1,953145.640.39.64
VWM dynamics model1.19B3,747138.537.913.84
Reviewer_vTpAOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary
  1. Modeling Comparison: Evaluating the performance and efficiency of the SCWM and DSWM paradigms.
  2. Explainable Evaluation: Utilizing explicit states to assess dynamics, policy bias, and uncertainty in hidden states.
  3. Data Augmentation: Leveraging LLMs to assist in building simulators, synthesizing diverse state trajectories to mitigate policy bias and introduce new actions.
Critical strengths
  1. Comprehensive experimental evaluation: Through multi-dimensional comparisons and ablation studies, the system systematically reveals the trade-offs between performance and efficiency for SCWM and DSWM.
  2. Interpretable evaluation: The system provides an in-depth characterization of model behavior by examining dimensions such as dynamical accuracy, policy deviation, and uncertainty in hidden states.
Critical weaknesses
  1. Insufficient analysis of the impact of state errors: SCWM relies on predicted states to drive rendering, yet the main text lacks targeted experiments to analyze how deviations in state prediction affect the final rendered output.
  2. Limited methodological innovation: The design of SCWM, separating state transitions from rendering, is similar to that of Agora-1. Furthermore, DSWM builds upon existing frameworks for joint multi-stream generation.
Questions for authors
  1. It is recommended to include state perturbation experiments to evaluate how varying degrees of state prediction deviation affect the final rendering results of the SCWM. Then, compare it with DSWM under the same perturbation conditions.
1 comment
AuthorsOpenReview ↗

Response to reviewer vTpA

Impact of state errors

State drift vs pixel drift (Table A). For each test clip, we generate two videos with the same SCWM renderer and the same random noise: One from the ground-truth game states and one from the states predicted by the SCWM's transition model. We then compare the two sequences (a) in pixel space using LPIPS and (b) in state space using the distance on object position. The main conclusion is that the renderer itself does not get worse over time and errors at long horizons are more likely to come from the drift in predicted states: The difference between the two videos samples grows during the first ~4 s (LPIPS 0.11 -> 0.39), but saturates at around 0.5 after that, while the distance in states keeps increasing throughout the rollout. In addition, the distance to the real video is about the same after 8–12 s (LPIPS 0.12) as in the first second (0.09).

Table A. State drift and pixel drift along SCWM rollouts measuring in error on object positions and LPIPS respectively.

HorizonDifference between the states: position RMSE (uu)Difference between the videos (LPIPS)Distance from ground-truth video to samples rendered from ground-truth states (LPIPS)Distance from ground-truth video to samples rendered from predicted states (LPIPS)
0–1 s1510.110.090.15
1–2 s3620.270.110.30
2–4 s8780.390.110.41
4–8 s2,0000.480.120.49
8–12 s2,7140.500.120.50

Sensitivity to state noise (Table B). We add Gaussian noise (with std σ\sigma ) to the positions of the ball or of all cars in the state space, and measure how it impacts gFID in two settings. Rendering: Noise is added to ground-truth states and the sequence is then passed through the renderer alone; and Rollout: Noise is added to every state the transition model predicts, so errors can accumulate.

Table B. Difference in gFID after adding noise to the state space (Δ gFID). σ\sigma is specified in unreal units (uu) here.

Noise onRenderingRollout
Ball, σ = 25+0.01+0.02
Ball, σ = 100+0.17+1.05
Ball, σ = 400+0.97+0.60
4 Cars, σ = 25−0.04+1.18
4 Cars, σ = 100+0.36+1.10
4 Cars, σ = 400+2.11+4.39

Limited methodological innovation

We fully acknowledge that some technical components we leverage have been individually explored in other world models. With this submission, our goal is not to introduce a new architecture but rather present a systematic study of integrating state-centric information in the standard video world models paradigm. As such we believe we provide the first in-depth analysis of SCWM/DSWM/VWM compared with matched training procedures. We also demonstrate additional features enabled by state-centric models which were not highlighted in previous work to the best of our knowledge: policy bias correction (Section 4.2), controllability via state manipulation (Appendix F1), uncertainty estimation (Section 4.3) and mixed-supervision training (Appendix F3). We have rephrased the introduction to make these aspects clearer. As for Agora-1, we did take note of the work but it was only released as a public blog post with little technical details (e.g. no results, architecture descriptions, states description etc.) which is why we only briefly mentioned it in related work.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Global Response

We thank the reviewers for their time, valuable feedback and insightful questions. Below, we first give a global response to issues highlighted by 2 or more reviewers. We further provide a detailed individual response to each reviewer and have uploaded an updated version of the main manuscript including some changes in red (changes include: better wording of the contributions, extended description of one related work, one additional latency plot as a new Figure 4).

Novelty and scope. With this work, our primary goal is not to introduce a new architecture for world models (WM) but rather present a systematic study of integrating state-centric information in the standard video world models paradigm. As such, even though SCWM (state-centric WM) and DSWM (dual-stream WM) share similarities with previous technical contributions, we believe we provide the first in-depth analysis of these architectures with equal training procedure along with a comparison to the classical VWM (video WM) paradigm. We also demonstrate additional features enabled by state-centric models which were not highlighted in previous work to the best of our knowledge: policy bias correction (Section 4.2), controllability via state manipulation (Appendix F1), uncertainty estimation (Section 4.3) and mixed-supervision training (Appendix F3). We have rephrased the introduction to make these aspects clearer.

Related work. Two previous works were mentioned by reviewers vTPa and HVid.

  • Agora-1 [18 May 2026] falls into the category of SCWM. We took note and cited this work but it was only released as a public blog post with little technical details (e.g. no results, architecture descriptions or states description) so it is difficult to place it in the literature. A follow-up, Agora-2, was released as a technical report on 21 September. Due to the release date we consider it concurrent work but there are nonetheless a few interesting observations to make

    • (i) Like us, Agora-2 leverages the RAE-like encoder from MIRA, which was released in between Agora-1 and Agora-2, suggesting this is a relevant design choice
    • (ii) The state representation in Agora-2 is very detailed: A state is described with 342 tokens and contains low-level game-specific information; For instance, 144 tokens are dedicated to the map, e.g. the appearance of the tiles. In contrast, we encode each explicit state with < 10 tokens mainly encoding generic information such as 3D positions of objects. This opens interesting questions for future work about what amount of state conditioning is necessary to build a good SCWM, especially in the neural renderer.
    • (iii) Several architectural choices in Agora-2 are tailored to this state space (e.g. dedicated predictions heads for subparts of the state) and not readily applicable to new games without heavy changes. In contrast, the only game-specific component of our architecture are the changes we make in RoPE position embeddings, and we provide ablation experiments on this component.
  • StatePlay [29 July 2026] shares similarities with our DSWM at a high level. We cited it in the previous version of the manuscript and have since expanded it in the related work section of the updated submission. The released StatePlay model is finetuned from a WAN2.2-5B model, with an additional separate stream for modelling states. The game states do not contain any position information but rather encode UI information of a fighting game (e.g. Player’s HP and attack gauge) with the underlying motivation to enforce mechanics consistency at the whole-game level (e.g. players die when their HP reaches 0). In addition, a core difference is that the noise level of the state stream and frame stream are tied, so denoising happens jointly: StatePlay only acts as a video world model with a latent state stream aiming to improve game mechanics consistency. In contrast, our DSWM samples two different noise levels for states and frames (Section 3.2), which unlocks two additional inference modes: Inverse renderer (clean video in, denoise state, see Section 4.3 about uncertainty prediction) and Frame renderer (state in, denoise frame). For the same reason, StatePlay can only be trained with paired state-frame data, while DSWM can be trained with mixed supervision (Appendix F3).

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.