acceptodds
Withdrawn from ICLR 2027

Injecting Causal Structure into Video World Models

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

World models should answer "what-if" questions about interventions on an environment. We introduce Causal Markov Supervision (CMS), a post-training procedure that injects a user-specified causal DAG into a video world model to improve intervention simulation. Each training episode is labeled with an event trace of DAG-variable outcomes and their timing, and a pre-trained video world model learns to predict traces along with predicts frames and actions after post-training. The DAG supervises temporal ordering and constrains attention so trace predictions satisfy its Markov property. In a realistic, AAA-quality Unity environment, confounding by player choice causes standard world-model training to fail on the causal effect of a "trapped chest” mechanic on level completion. CMS accurately simulates identifiable intervention outcomes without degrading pretrained visual fidelity. With intervention labels, it also exploits intervention-induced independences to simulate beyond the observational training distribution. Finally, we show that a vision–language model can automatically label training episode with DAG event traces.

Primary area:
causal reasoning
Keywords:
world models, video world models, causal inference, causal representation learning, structural causal models, simulation, gaming, world action models, vision-language models, multi-modal models, inductive bias

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_yjpROpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper addresses the failure of video world models to distinguish observation from intervention under confounding. It proposes Causal Markov Supervision (CMS), a post-training procedure that injects a user-specified causal DAG into a pretrained video world model. The intuition is that the DAG’s Markov property specifies which history is irrelevant given the evidence. Penalizing disagreement between paired predictions that differ only in such history therefore teaches the model what not to attend to. In a single synthetic Unity environment, the arms trained with intervention data estimate the target intervention outcome reasonably well with little loss in visual fidelity. However, the Markov loss itself does not improve the primary metrics, and the full CMS model recovers causal structure worst among all arms.

Critical strengths
  1. Well-motivated problem and novel formulation: The DAG-forward workflow and the use of paired-prediction JSD to enforce DAG-implied conditional independences offer a simple way to inject causal inductive bias into a video world model.
  2. Targeted evaluation design: The setting where observational and interventional contrasts have opposite signs directly tests whether a model conflates P(R | do(X), A) with P(R | X, A).
  3. Practical trace labeling: GPT-5.6 Terra labels 99.3% of event values correctly which suggests that DAG traces can be obtained without manual annotation.
Critical weaknesses
  1. Markov supervision does not help: Adding the Markov loss increases Δγ̂₁ from .125 to .196 without interventions and from .003 to .063 with interventions (Table 4).
  2. Claims contradicted by results: The claim that “CMS recovers causal structure” conflicts with Table 4 where CMS has the worst SID (44) and d-separation F1 (.320).
  3. No evidence that standard training fails: The paper reports no baseline that produces the wrong-sign contrast on the do(X) query even though the abstract states that confounding causes “standard world-model training to fail on the causal effect”.
  4. Trace–video inconsistency: Generated traces agree with the generated videos on only 35–61% of outcome values (vs. 99.7% on simulator video) which undermines the claim that CMS “accurately simulates identifiable intervention outcomes”.
  5. Limited experimental scope: All results come from one synthetic environment with a single nine-variable DAG and a single query and without multiple random seeds.
Questions for authors

see weakness

Reviewer_jagrOpenReview ↗
Rating
1
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

Causal Markov Supervision (CMS) is a post-training procedure for pretrained causal video world models. A user-specified DAG is injected by (i) extending the model to predict a per-episode DAG trace (each variable's realized value and time) and (ii) adding a Markov loss comparing paired predictions whose histories differ only in components the DAG deems conditionally independent. Interventions use graph surgery. Evaluation is in one synthetic Unity chest-room with a ground-truth SCM: the effect of chest-glow X on reaching the good exit R, confounded by player curse Z and trap W, with observational and interventional contrasts of opposite signs (Table 2).

Critical strengths
  • Well-motivated target: logged-trajectory world models miss counterfactuals under changed mechanisms, and the Table 2 sign-flip makes the case concrete.
  • DAG-trace supervision and a swap-style Markov loss encodes conditional independences into an existing model without retraining.
  • The VLM auto-labeling method (Table 3) is a useful calibration where it lands near-perfect trace values (99.3%) but weak DAG structure (SID 15, d-sep F1 .53).
Critical weaknesses
  • The proposed method is not best on any metric in the paper's own table, and the ablation is not at matched training. CMS is at Step 100 while Trace, Trace+Markov and Trace+Intervention are all at Step 500. On the primary metric CMS has Δγ^1\Delta_{\hat{\gamma}1} = .063 against .003 for Trace+Intervention, and CMS is also worst on the joint JSD (.237 against Trace's .103), second on trace-video agreement (.538 against .613), and worst on both structure metrics (SID 44, d-separation F1 .320, against 18 and .578 for Trace+Markov).
  • The evaluation is one environment, one nine-variable DAG, one query and one intervention target, with no external method to compare against. A forward DAG workflow is being argued from a single instance. Misspecification is the obvious failure mode for a method that injects a user graph, and the limitations paragraph raises it without testing it.
  • Final sentence in contributions: "CMS recovers causal structure and predicts intervention outcomes under strong confounding while preserving real-time, high-fidelity video generation." Table 4 shows the opposite ordering as components are added: SID moves 19, 18, 28, 44 and d-separation F1 moves .346, .578, .403, .320 across Trace, Trace+Markov, Trace+Intervention and CMS, so the Markov term is the only addition that helps and the full method is worst. A VLM reading real simulator video also infers a better DAG (SID 15, F1 .530) than any post-trained arm's generated traces imply. "Recovers causal structure" should be dropped or explained.
Questions for authors
  • Can you report the ablation with all arms at the same checkpoint? CMS at Step 100 against everything else at Step 500 leaves the central comparison open. What is CMS's Δγ̂1 at Step 500, and if 100 was chosen because it is CMS's best step, does the method degrade with more post-training?
  • What happens when the supplied DAG is wrong? A missing confounder, a reversed edge or a spurious edge is the realistic failure mode, and one perturbed DAG would change how I read the workflow claim.
Reviewer_yGCDOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary
  • this paper introduces causal markov supervision (CMS), a training framework that incorporates a user-specified causal DAG into a video world model to simulate interventions.
  • the core idea is extending a block-causal transformer backbone to jointly predict chronological event traces and video frames. timing is modeled via a raw hazard function with a structural gate enforcing parent-child DAG ordering, while a markov loss penalizes dependence on history blocks that are conditionally independent according to d-separation in the DAG.
  • the proposed method is evaluated on a synthetic Unity dungeon crawler environment with 9 binary variables across 28-second clips. their analysis suggests that CMS improves intervention outcome simulation over purely observational training
Critical strengths
  • causal modelling in videos is of very broad interest in the research community which this paper attempts to tackle.
  • this paper aims to create a controlled experimental setup towards developing an alternative pipeline which is also appreciated.
Critical weaknesses
  • the submission title on openreview does not match the full title on the pdf.
  • the uploaded supplementary material is just a duplicate copy of the main paper instead of the actual appendix.
  • the paper points to the appendix 23 separate times for key details, but none of that appendix is accessible to reviewers.
  • the dataset lacks real-world diversity and complexity. it is limited to a single synthetic 28-second Unity dungeon room with only 9 simple binary variables.
  • the paper mentions a 171m-parameter world-action model, tokenizers, and a bridge-flow squared error, but it never explains how frames are tokenized or the details of the error formula.
  • the paper relies heavily on video game jargon (like chest rooms, curses, wards, and loot mechanics), making it harder to read.
  • there is a contradiction in the paper claim, even when GPT-5.6 Terra labels 99.3% of the event values correctly, the causal DAG still gets a very poor structural intervention distance (SID of 15/12) and a low F1 score (.530/0.416, which is worse than random). clearly this is not a stable pipeline for data preparation.
  • the paper's proposed CMS, which is actually outperformed by a simpler ablation (trace + intervention) in table 4 on ∆ (.063 vs .003) and ρ3 (.538 vs .613).
  • full CMS gets an SID of 44 and an F1 of .320, which is far worse than the basic trace model (SID 19, F1 .346) and the trace+markov model (SID 18, F1 .578). what is the purpose of CMS then?
  • while all baseline models are evaluated at checkpoint step 500, CMS is evaluated at step 100 with no explanation, why so? did it diverge due to longer training?
  • the VLM agrees with generated video frames only 35% to 61% of the time, meaning the visual frame generator and the causal trace head frequently contradict each other.
  • the paper does not compare against any standard, existing world model baselines from literature.
Questions for authors

please see weaknesses

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.