- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 1
- Soundness
- 2
Read review
Paper summary
This paper proposes a paired-intervention evaluation framework for assessing whether action-conditioned world models correctly predict how future states change under different actions. The authors introduce pixel-level causal alignment, response magnitude, and normalized error metrics, using deterministic simulators to obtain exact counterfactual ground truth. Experiments on three world model architectures across 26 Atari games reveal that models with similar prediction accuracy can exhibit substantially different causal responses.
Critical strengths
- The paper addresses an important evaluation problem by highlighting the limitations of conventional prediction-quality metrics in assessing action-conditioned world models.
- The proposed paired-intervention protocol provides a systematic way to compare predicted action-induced changes against exact counterfactual outcomes from simulators.
- The experiments across multiple world model architectures and Atari games provide empirical evidence that similar prediction accuracy can correspond to substantially different causal alignment.
Critical weaknesses
W1. Insufficient Validation of the Proposed Metric
The proposed metric measures pixel-level differences rather than semantic or decision-relevant action consequences. Small spatial misalignments or irrelevant visual changes may significantly affect alignment without reflecting actual failures in action understanding. Although the authors attempt object-level validation, the automated readout fails human validation, leaving the construct validity of the proposed metric insufficiently established.
W2. Limited Novelty and Unclear Advantages Over Existing Methods
Prior works such as ScratchWorld, Genie, ACT-Bench, and iWorld-Bench have already investigated counterfactual state changes or action controllability beyond visual quality. The paper does not sufficiently clarify its methodological novelty relative to these approaches. Moreover, the proposed metric is highly correlated with Genie's ΔPSNR, and its additional diagnostic value is not convincingly demonstrated on real trained world models.
W3. Limited Practical Relevance and Poor Presentation
Despite motivating the metric through planning and decision-making, the experiments fail to establish a convincing relationship between causal alignment and downstream performance. Some low-alignment models still achieve substantial planning gains, raising questions about the metric's practical significance. Furthermore, the paper is extremely difficult to follow, with critical methodological details and related work largely relegated to the appendix and numerous auxiliary analyses obscuring the central contribution.
Questions for authors
Regarding W1: Insufficient Validation of the Proposed Metric
Q1. Can the authors validate whether pixel-level causal alignment correlates with the correctness of action-induced object movements or other semantic state changes?
Regarding W2: Limited Novelty and Unclear Advantages Over Existing Methods
Q2. Please clarify the methodological novelty of the proposed approach compared with existing counterfactual and action-conditioned world model evaluations, particularly ScratchWorld [1], Genie [2], ACT-Bench [3], and iWorld-Bench [4].
Q3. Can the authors demonstrate concrete failure cases on real trained world models where the proposed metric detects meaningful errors that existing metrics, particularly Genie's ΔPSNR, fail to identify?
Regarding W3: Limited Practical Relevance and Poor Presentation
Q4. Better to provide a more controlled evaluation using simulator-native rewards to demonstrate whether the proposed metric predicts downstream planning performance better than PSNR and ΔPSNR
Q5. Better to improve the organization and logic for the writing
[1] Lin, Y., & Zhang, J. (2026). ScratchWorld: Evaluating If World Models Compute Executable Consequences. arXiv:2606.31689.
[2] Bruce, J., et al. (2024). Genie: Generative Interactive Environments. ICML 2024.
[3] Arai, H., et al. (2024). ACT-Bench: Towards Action Controllable World Models for Autonomous Driving. arXiv:2412.05337.
[4] Fang, J., et al. (2026). iWorld-Bench: A Benchmark for Interactive World Models with a Unified Action Generation Framework. ICML 2026.
4 comments
To Reviewer Ya7M: our answer in brief
We thank the reviewer again for the review. Our full answer is in our two comments under this review. The discussion period is about to close, so we summarise that answer here in five points, each with its place in the revised PDF.
- W1, Q1 (validation). Human labels that do not use the automated locator agree with the alignment classes. The annotators judged the movement of the actuated object correct in 9 of 11 tracked panels and in 2 of 13 never-aligned panels, and a tolerance of up to 4 pixels lifts at most 2 of the 28 never-aligned cells to their threshold (Section 4.3; Tables 27 and 28, Appendix V).
- W2, Q2 (novelty against the four cited works). Section 1 now compares the measurement with ScratchWorld, Genie, ACT-Bench and iWorld-Bench, and Table 5 (Appendix A) lists all four. Ours is the only one of these that compares a trained pixel world model's change under an action intervention with the exact counterfactual change of the world.
- W2, Q3 (value over ΔPSNR on trained models). ΔPSNR's interval lies above zero in eight cells where alignment is below 0.05. In the seven with a measurable ΔPSNR, 40% to 100% of the model's change lies where the world does not change (Section 4.5; Table 29, Appendix W). Part 1 of our answer describes one of them, STORM on Boxing.
- W3, Q4 (downstream performance). Two tests, each under a rule written before the runs, use the emulator's own game score and the emulator's own frames. The gain of reward-ranked planning follows PSNR (ρ = 0.70), and success at reaching a goal follows alignment (0.89, against 0.61 for PSNR; Section 4.6, Figure 4; Appendices X and Y).
- W3, Q5 (presentation). The introduction now states the central contribution and the comparison with the four works, and ends with Table 1, a summary of the findings that names the section for each. Figure 4 shows both tests of usefulness, and the secondary analyses moved to Appendices F, K and N.
We should be most grateful if the reviewer would tell us whether these answers resolve the concerns, and would consider them in the final assessment.
Thank you for clarification and additional experiments. The rebuttal has addressed most of my concerns including metric validity and methodological differences.
The revised paper still does not establish a coherent progression from the research question, to the limitations of existing methods, to the proposed solution and what specific gaps it addresses. Instead, the introduction presents four results that are difficult to understand without reading details throughout the paper. This weakens the overall clarity of the scientific contribution and readability.
Overall, I appreciate the substantial improvements. I also recognize the value of the proposed diagnostic framework from the perspective of ''direct measurement of this two-trajectory quantity''. However, the central scientific motivation and the general significance of the work remain insufficiently justified with the current paper logic. I therefore maintain my original rating.
Reply to Reviewer Ya7M: the progression from research question to contribution
We thank the reviewer for the reply, for confirming that our response addresses the concerns on metric validity and methodological differences, and for recognising the value of the direct measurement of this two-trajectory quantity. The remaining point concerns the logic of the paper. We have revised the introduction to follow the progression that the reviewer describes: from the research question, to the limitations of existing methods, to the proposed solution and the specific gaps it addresses. The introduction now also defines alignment and the cell classes before it states any result, so the four results can be read without the later sections. The revised PDF, uploaded with this reply, contains the new introduction. The revision reorders existing text and adds no new result, and every section, table and figure number is unchanged.
The progression, with its place in the revised PDF.
- Research question (Section 1, paragraph 1). A world model exists to answer a planner's counterfactual question: if the agent took this action instead of that one, how would the future differ? Uses that read the predicted consequence, such as reaching a goal image or asking what an action does, need this answer to be right. The paper asks whether trained world models get it right: when the action changes, does the model's future change the way the world's does?
- Limitations of existing methods (paragraphs 2 and 3; Section 2). Current evaluations score one generated trajectory at a time, for fidelity, plausibility or memory, whereas a correct response is a property of two trajectories that differ in one action. Proposition 1 shows that no action-blind score of one rollout, and no score of the distribution of rollouts, can measure it. Corollary 1 shows that PSNR certifies it only when the prediction error is small compared with the true effect, which, under an equal-error assumption, fails in 23, 24 and 25 of the 26 games for the three models. The closest evaluations that do change the action give no exact counterfactual for a pixel world model: Genie's ΔPSNR compares the two rollouts only with the factual future, ACT-Bench and iWorld-Bench score one generated video, and ScratchWorld concerns language models that predict symbolic state (Appendix A, Table 5).
- Proposed solution (paragraph 4; Definition 1 and Section 3). The emulator is replayed under both action sequences from the same state, so the world's change is exact. The model's change is compared with it as an alignment, together with a magnitude and an error. This is the direct measurement that the reviewer names.
- Gaps addressed (the four results). Paragraph 4 ends by naming the four results, and Table 1 lists each with its evidence.
- What accuracy misses. DIAMOND predicts Alien within a decibel of Pong, yet its alignment at delay 20 is 0.002 on Alien against 0.86 on Pong, and 28 of the 78 model-game cells never align (Sections 4.1 and 4.2).
- What a low alignment means. It means a wrong consequence, not a small offset. Human labels separate the classes, a tolerance of up to 4 pixels lifts at most 2 of the 28 never-aligned cells to their threshold, and no measured delay lifts any (Section 4.3).
- What ΔPSNR misses. ΔPSNR's interval lies above zero in eight cells where alignment is below 0.05, and 40% to 100% of the model's change in the seven with a measurable ΔPSNR lies where the world does not change (Section 4.5).
- Which score predicts usefulness. Success at reaching a goal frame follows alignment (ρ = 0.89, against 0.61 for PSNR), and the gain of reward-ranked planning follows PSNR (Section 4.6, Figure 4).
Motivation and significance. A world model is used to choose and to explain actions, and both uses rely on its predicted consequence of an action. The scores used to select models cannot certify that this consequence is right (Proposition 1, Corollary 1). The paper shows that the failure is common in trained models, that accuracy does not reveal it, and that it matters where the consequence is used. Chance-corrected success at reaching a goal frame has a median of 0.11 in never-aligned cells against 0.62 in tracked cells (post hoc; Table 33). The measurement needs only an environment that can be replayed exactly and two model rollouts per episode, so it applies to any world model trained in a simulator or a game.
We hope that the revised introduction makes the logic of the paper clear on a first reading. We should be most grateful if the reviewer would consider the revision, together with the concerns now addressed, in the final assessment.
Hi, just wanted to check if you are happy with our rebuttal or if you have any further concerns. The deadline is approaching, so it would be helpful to hear your thoughts. Thanks!