acceptodds
Under review as a conference paper at ICLR 2027

Empowering World Models with Deconfounded Dynamics via Multi-Level Causal Alignment

Abstract

Implicit world models predict future visual representations without reconstructing images, supporting tasks such as embodied planning. However, encoding object and background information into a shared feature space can entangle task-relevant states with incidental visual context. This may bias future-state predictions and propagate errors over successive predictions. To address this problem, this paper presents a Deconfounded Causally Aligned Latent Model (DeCALM), which empowers world models with deconfounded dynamics by reducing background dependence in latent encoding. Specifically, DeCALM comprises two main modules: Background-Deconfounded Latent Encoding (BLE) and Multi-Level Causal Alignment (MCA). BLE uses object and background representations separately, approximates backdoor adjustment with a background dictionary, and constrains predictions under counterfactual background changes. MCA takes the resulting features and adaptively combines absolute-state and incremental predictions to improve future-state prediction. Experiments on three benchmark datasets show that DeCALM achieves the highest success rate among the compared methods, including a 14.58-percentage-point gain over the strongest baseline on Drawer Open. Further analyses suggest that DeCALM mitigates the ill-posedness of latent representations under visual disturbances, yielding more task-relevant features and more accurate state-change predictions at later horizons.

Primary area:
unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords:
Implicit World Models, Embodied Planning, Causal Representation Learning, Backdoor Intervention

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_8hCvOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes DeCALM, a framework designed to improve the robustness of object-centric latent world models against visual background disturbances, built on causal-jepa. The method introduces two main components: BLE and MCA. BLE constructs a dictionary of background prototypes from training data and integrates these prototypes into the latent representation to approximate backdoor adjustment. MCA combines next-state prediction with incremental state change prediction through a gating mechanism, aiming to reduce error accumulation during long-horizon rollouts. Experiments on 3 manipulation tasks from OGBench and MetaWorld show improvements over world model baselines.

Critical strengths
  • Meaningful performance improvenemt : The paper addresses an important challenge in visual world modeling: background disturbances can affect latent predictions and consequently degrade planning performance. Authors' results suggest that the proposed approach can improve the practical robustness of latent world models under the evaluated visual disturbances.

  • Practical integration with pretrained world models and supporting analysis : DeCALM builds upon a causal-jepa model while keeping the visual encoder, slot-attention module, and dynamics backbone frozen. This makes the approach potentially useful for improving existing world models without retraining the entire architecture.

Critical weaknesses
  • The causal claims seem stronger than what the method actually supports : I believe the paper makes somewhat strong claims about "deconfounded dynamics," including in the title. The proposed method mainly addresses background dependence in latent predictions, but this does not necessarily mean the learned dynamics are deconfounded in a broader causal sense. I also find the use of the term "counterfactual" somewhat misleading. The method replaces background tokens while keeping object representations and future targets fixed. This seems closer to prediction consistency regularization under background substitution than actual counterfactual reasoning. I believe there is also a gap between the backdoor adjustment in Eq. 4 and its implementation in Equation 6-8. Averaging background features before prediction is not generally equivalent to averaging predictions over background conditions, especially with a nonlinear predictor. While their Appendix B discusses the idealized adjustment, it remains unclear how closely the actual feature-level implementation approximates it.

  • The method relies on fairly strong assumptions that should be discussed more explicitly : The method assumes that background information can be separated from object representations and summarized using a fixed number of prototypes. However, background changes do not always affect only the background slots. For example, changes in lighting (as authors mentioned in the paper) can affect the entire observation, including object appearances, shadows, and boundaries. In such cases, keeping object representations fixed while changing background tokens may not be a realistic assumption. Also, all baseline models that they compare with do not use the same background-specific information (or SCM). I think it makes it difficult to disentangle the benefits of the proposed causal formulation from those of the additional background-specific inductive biases.

Questions for authors
  1. Could the authors clarify why incremental state prediction is considered a form of causal alignment? From the current formulation, it seems similar to residual next-state prediction with an auxiliary loss. What makes this component specifically causal, rather than a standard architectural choice? Could the improvements be explained by the skip connection or additional supervision?

  2. Could the authors provide a comparison with simpler alternatives, such as removing the background token or randomly replacing it during training? This would help clarify whether the improvements come from the proposed causal adjustment or more general background invariance regularization.

  3. Could the authors compare the proposed feature-level prototype integration with the prediction-level backdoor adjustment in Eq. 4? This would help clarify whether the approximation captures the intended causal effect or mainly acts as a context adaptation mechanism.

Reviewer_GwkKOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
3(+2)
Read review
Paper summary

The paper proposes DeCALM, a method intended to make visual world models less sensitive to irrelevant backgrounds and more accurate during multi-step prediction for robotic planning. It builds on Causal-JEPA, keeping the visual encoder, object-slot extractor, and pretrained dynamics backbone frozen while training additional modules. Its background module uses a dictionary of background features and background-token substitutions to encourage consistent predictions across changes in visual context. Its dynamics module learns to combine direct next-state predictions with predictions of incremental state changes. Experiments on three manipulation tasks report improved planning success, including a 14.58-percentage-point gain over the strongest baseline on Drawer Open.

Critical strengths

This paper addresses an interesting problem, ie irrelevant visual backgrounds can distort dynamics predictions and downstream planning. The paper targets robustness to these distractions at both the representation and prediction levels. DeCALM improves planning success across three manipulation tasks, including a 14.58-percentage-point gain over the strongest baseline on Drawer Open. Component ablations provide additional evidence that the proposed additions contribute to performance. The solution uses frozen models which can potentially be adapted to any world models, though currently only with Causal-JEPA.

Critical weaknesses

The proposed intervention does not fully address the background contamination motivating the paper. To test the robustness claim directly, the background should be changed in the input images, followed by re-encoding the entire image. Currently the method changes only the background token after encoding. This matters because deconfounding is a central contribution, yet the evidence does not distinguish it from ordinary feature adaptation and consistency regularization.

The proposed Multi-Level Causal Alignment is not clearly distinguished from an ordinary residual predictor. Claiming causal alignment requires additional justification. A residual-only head and a fixed mixture would be essential comparisons. This limits the methodological contribution. The reported improvement could come from a residual connection or added capacity rather than the claimed causal alignment.

Questions for authors

Please answer the points in the weakness section.

Reviewer_WtKpOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

JEPA models often confuse the regions informative for the executed action and as a result base their future predictions on irrelevant background features. This work aims to robustify the JEPA transitions against background changes by breaking the dependence of observation from the background in the causal formulation. This is practically achieved in a new training architecture that robustifies the method against collections of background features from the training set. The resulting method outperforms the baseline on robotic manipulation tasks and shows qualitatively its robustness to background changes.

Critical strengths

The work defines an interesting and important problem of decoupling in the representation the decision-making from the background The proposed architecture is shown experimentally to improve the overall performance (over Causal JEPA) and qualitatively to be robust to background change. The method's architecture is theoretically grounded rather than purely build on empirical results.

Critical weaknesses
  • MCA relevance to the problem - What is the role of Multi-Level Causal alignment in the context of the claimed contribution of the paper - decoupling the background? While MCA has shown to improve the results empirically, it is not clear why predicting gated absolute and relative transitions is helping and how it is related to the background hypothesis. As Lcf and Linv give quite a boost of the final model, it would be helpful to see in the ablation the results of Causal-JEPA + BD + CE (without MCA) to establish the performance of the background handling machinery introduced in the paper.
  • Analysis concerns - All analysis in 5.4 and Fig.3 seem to base the conclusion on very minor changes in the metric. For b and c the metrics were multiplied by 10^3 in order to boost the distances between the curves. Cosine simularity difference is less than 0.0005 in the figure. Based on these differences conclusions are made about improvements. In order to be more convincing both 1) the statistical significance has to be established (e.g. p-value of t-test) and 2) the practical significance of the improvement has to be demonstrated. For 2), it is possible that a statistically significant tiny improvement in latent space might translate to large improvement in a task, or it might make no positive difference in practice. Latent space difference with MSE does not give us a clear sense of scale and how that translates out of the latent space, as the scale can also change also between retrainings of the (now frozen) JEPA encoders. I would suggest the authors to at least provide percentage of improvement over the baseline over time (precise numbers) instead of MSE and judge if this improvement is significant.
  • Background robustness evaluation - The two main lines of evaluation in the experiment section are solving the benchmark tasks as they are intended and proving that handling backgrounds can improve performance in the general case, and demonstrating the specific benefit of being able to swap out backgrounds without significantly dropping performance. While the first point is well shown in Table 1 and 2, the second point is only demonstrated through qualitative examples. While there are multiple examples - in appendix and in main paper, it is necessary to establish these results quantitatively as well - by performing and experiment where the setup of randomly swapping backgrounds on the history on many samples, measuring the baseline's error and the proposed method's error with std (both on the same samples). This would significantly strengthen the paper.
  • Limitations - Limitations of the method are not stated clearly and the setup suggests such could exist - potential inability to extend to unseen backgrounds in training, inability to handle non-static backgrounds, scalability problems to a large number of backgrounds, reduced usability in complex and real-world backgrounds. It is important to establish what this method currently allows and what it does not.
  • Source of improvement in CE - One observation is that Lcf leads to a rather minor improvement while Linv leads to a major one. It appears that the contrastive objective by itself is what generates value but aligning it to ground truth does not help much. It would help to find an explanation of why that is.
  • Writing Clarity - The work can use reworking to improve clarity - the excessive use of abbreviations makes it hard to follow and requires cross-referencing through text. E.g. in the main figures abbreviations are used that are no defined in the figure title or anywhere in text - they have to be inferred from subsection titles.
Questions for authors

Main questions:

  • How does MCA contributing in specific to the background separation goal established in the paper? What is the motivation for the two branch approach at the heart of MCA?
  • What is the statistical significance of All experiments in 5.4/Fig.3 (without multiplying by a factor)? What is the practical significance of these differences in terms of scale?
  • How do you show quantitatively the claimed background robustness of your method?
  • What are the limitations of the method? Can the method extend to backgrounds never seen in the training set or it is limited to training backgrounds only? Is the method only applicable for static backgrounds and still camera? Is the method scalable to a large number of varying backgrounds? Is the method applicable to real-world visually complex static backgrounds?
  • What is the source of improvement in CE?

Additional questions:

  • It appears that CE training requires a separate protocol - how does that work in practice - do you jointly construct counterfactual sample for every batch element or is it a separate sample in the training?
  • What is masked history reconstruction? What are historical object representations?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.