- Rating
- 2
- Confidence
- 2
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper investigates how explicit physical state information can improve video world models. The authors explore two architectures: a State-Centric World Model (SCWM), which decouples physical state transitions from video rendering, and a Dual-Stream World Model (DSWM), which jointly models video and physical states through cross-attention. Additionally, the authors introduce RoLeSynth, an LLM-assisted synthetic data generation approach to mitigate policy bias and improve behavioral diversity. Experiments on the Rocket Science dataset demonstrate that SCWM achieves better modeling efficiency and video generation quality, while DSWM enables uncertainty estimation of partially observed states. The authors also validate the effectiveness of synthetic data augmentation in reducing policy bias and improving generalization across different behavior distributions.
Critical strengths
-
Strong empirical results and efficiency improvements. The proposed SCWM achieves competitive video generation quality while significantly reducing the computational cost of dynamics modeling. The experiments demonstrate that a small state transition model can achieve performance comparable to much larger video world models, highlighting the efficiency benefits of explicit state modeling.
-
Systematic exploration of state-based world modeling. The paper investigates two different architectures, SCWM and DSWM, and provides detailed experiments and ablation studies to analyze their performance and trade-offs. The DSWM also demonstrates interesting capabilities in estimating hidden physical states and their uncertainty.
-
Interesting analysis of policy bias and data augmentation. The authors identify policy bias in the Rocket Science dataset and introduce RoLeSynth to generate diverse synthetic state trajectories. The experiments provide useful insights into the distinction between learning physical dynamics and learning behavior distributions, and demonstrate the benefits of synthetic data augmentation for generalization.
Critical weaknesses
-
Unclear research motivation and methodological contributions. The paper investigates several directions, including state-centric world modeling, dual-stream generation, uncertainty estimation, and LLM-assisted data augmentation. However, these directions appear relatively independent, and similar ideas have already been explored in previous works. It remains unclear what specific research problem the paper primarily aims to address and what new methodological contributions or generalizable insights it provides beyond existing approaches. A clearer motivation and a more focused discussion of the key contributions would strengthen the paper.
-
Insufficient comparisons with existing state-aware world models. The main experiments primarily compare the three architectures implemented by the authors, while closely related approaches such as Agora-1 and StatePlay are not directly evaluated. Given the similarities between these methods and the proposed SCWM and DSWM, more direct comparisons or controlled architectural ablations would help establish the independent contributions of this work.
-
Potential confounding factors in the experimental evaluation. The SCWM transition model is trained with approximately nine times more state-only data, and SCWM relies on explicit initial state information, whereas VWM primarily uses visual observations. These differences make it difficult to determine whether the reported improvements come from the proposed architecture, additional training data, or privileged state information. More controlled experiments under comparable training and input settings would strengthen the conclusions.
Questions for authors
N/A
3 comments
I appreciate the authors' detailed rebuttal and additional experiments, particularly the controlled comparisons of training budgets and initial state conditions. These results help clarify several of my experimental concerns.
However, my primary concern remains the limited methodological novelty and generalizable insights. While the authors position the paper as a systematic study rather than a novel architectural contribution, I believe such a study should establish meaningful and broadly applicable conclusions beyond a single experimental environment.
In particular, the additional experiments show that SCWM's advantages depend substantially on access to accurate ground-truth states, with its video generation performance deteriorating when states are estimated from visual observations. This raises an important question about the conditions under which state-centric modeling is actually preferable to video-based modeling.
Furthermore, although the authors clarify the differences from existing state-aware world models, the independent contributions over prior approaches remain insufficiently established.
Overall, I acknowledge the empirical value of the work, but I do not find that the current study provides sufficiently novel or generalizable insights to justify acceptance at ICLR. I will therefore maintain my original rating.
Thank you for your response and acknowledgement of the rebuttal; We would like to address three short points below.
“Furthermore, although the authors clarify the differences from existing state-aware world models, the independent contributions over prior approaches remain insufficiently established."
Beyond the systematic study, we have restated additional contributions in the general response (policy bias correction (Section 4.2), controllability via state manipulation (Appendix F1), uncertainty estimation (Section 4.3) and mixed-supervision training (Appendix F3)) and differences with recent related work (Agora-1, and mixed supervision setting DSWM in Appendix F3 as compared to StatePlay). We would be grateful if the reviewer could point to the specific contributions they find insufficiently supported, so that we can address them directly.
“This raises an important question about the conditions under which state-centric modeling is actually preferable to video-based modeling."
We respectfully disagree that this is a weakness of our work specifically: this is a criticism of state-aware models in general. Moreover, there is a growing body of work showing benefits of state-aware modelling, for instance for controllability or efficiency in planning/simulation applications. Here is a quick selection of recent work on games and robotics:
- Ye et al. PointCast: One World Model for Rigid, Articulated, and Deformable Object Manipulation. Sep 2026.
- Deng et al. WorldMind: Decoupled Game World Model for State-Aware NPC Behavior. Aug 2026.
- Meng et al. Marionette: Predicting World States, Rendering Geometry, Painting Appearance. Aug 2026.
- Cai et al. MASS: Multiplayer World Models with Authoritative Shared State. Aug 2026.
- Zhang et al. World Action Planner: Generalizable Robot Decision-Making with Action-Conditioned World Models. Jul 2026.
- Kim et al. Robot-Factored World Models via Robot Rendering. CoRL 2026
- Li et al. From Pixels to States: Rethinking Interactive World Models as Game Engines. Jul 2026.
“SCWM's advantages depend substantially on access to accurate ground-truth states…”
We agree, but this result illustrates the core motivation of SCWM: they can leverage states to further improve model performance, which VWM can not by design. Judging SCWM on estimated, rather than ground truth, states is out of distribution as it was only trained on actual states, thus cannot compensate for any error of the inverse renderer. In addition, the required ground-truth state prompt is not necessarily a difficult-to-achieve requirement (e.g. see below a quick study where we measure both models at different prompt lengths, showing we need as little as 4 frames, without severe degradation to visual quality).
| Context given | VWM (video) gFID ↓ | SCWM (states) gFID ↓ |
|---|---|---|
| 4 s | 10.37 | 9.62 |
| 2 s | 10.36 | 9.52 |
| 1 s | 10.54 | 9.63 |
| 0.4 s | 10.53 | 9.82 |
| 0.2 s (4 frames) | 10.42 | 9.73 |
Response to reviewer Hvid
Unclear research motivation and methodological contributions
Our goal is not to introduce a new architecture for world modelling but rather present a systematic study of integrating state-centric information in the standard video world models paradigm in a controlled setting. As such, even though SCWM and DSWM do share similarities with previous technical contributions, we believe we provide the first in-depth analysis of these architectures with equal training procedure. We also demonstrate additional features enabled by state-centric models which were not highlighted in previous work to the best of our knowledge: policy bias correction (Section 4.2), controllability via state manipulation (Appendix F1), uncertainty estimation (Section 4.3) and mixed-supervision training (Appendix F3). We have updated the introduction accordingly.
Potential confounding factors in the experimental evaluation
Data Size. We would like to clarify that the SCWM is trained on nine times more steps but not more data: The transition model of the SCWM train on more epochs of the same dataset. This is possible because the transition model is efficient to train (45 GPU hours vs 423 for the VWM), which we see as a strength of SCWM. We also evaluate the SCWM at earlier training stage of the transition model to match the number of samples seen in training (
Table C). Overall, training the VWM for longer does not improve its visual quality. The same is true for SCW but the state prediction does improve with more training.Table C. Matched data budgets.
States as prefix. Thanks, this is an interesting question.
Table D, we estimate (a) the SCWM with states prefix estimated from frames (rather than ground-truth) and (b) the VWM with prefix rendered from ground-truth states (Table E, top rows) . The results show the VWM does not benefit from knowing ground-truth states, while SCWM does exploit this information.Table E(bottom rows) including states in the prefix generally leads to better visual quality (both for the SCWM and DSWM) which shows state-centric information is also well exploited by DSWM.Table D. Changing the modality of the 4 s ground-truth prefix.
Link to Agora-1 and StatePlay
We have extended our citations of both work in the updated manuscript to provide additional context. We give a short summary comparison below and add a more detailed commentary in the global response above.