- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper studies whether world models can learn representations that correspond to underlying causal variables. It formulates the problem using a dynamical system with latent causal states and proves that such representations can be learned under invertibility and other assumptions. Motivated by this, this paper proposed the method action-modulated JEPA (A-JEPA). Extensive experiments show the effectiveness of this method over multiple tasks.
Critical strengths
- The authors formulate a model to analyze latent causal dynamics and introduce assumptions like invertibility to characterize that the learned representations could match the underlying causal variables. Inspired by this theory, they propose an Action-Modulated additive-noise model and yield A-JEPA from this.
- The authors implement controlled experiments to show the proposed method's capabilities in latent-state recovery, latent-structure recovery, and visual dynamics prediction.
Critical weaknesses
- Causal representation learning is a important topic. However, its importance for dynamic prediction is unclear: The causal representations discussed in the paper may be equivalent, through an invertible transformation, to representations obtained through conventional training, without a loss of information. Therefore, the additional contribution of causal representations to our understanding of world models remains unclear.
- Beyond controlled environments, the experiments do not sufficiently show whether the model learns causal representations or why it learns these representations in real-world settings. Additional experiments in more realistic settings or further ablation studies would help clarify these points.
Questions for authors
- Although component-wise causal identifiability is theoretically stronger than information preservation up to an arbitrary invertible transformation, its practical benefits remain unclear to me. Could the authors clarify what additional advantages causal representations provide for world models compared with equally information-preserving representations?
- In Figure 5, the transfer performance is evaluated through latent rollout prediction. Could the authors clarify to what extent these results demonstrate the recovery of underlying causal mechanisms, rather than improved predictive representations? Would experiments involving controlled interventions provide more direct evidence?
3 comments
Thanks for raising this concern. We would like to clarify that, in real-world environments, the underlying causal variables and mechanisms are generally unknown, making their direct evaluation challenging. Similar to existing work in causal representation learning community, we therefore consider indirect evaluations that examine properties expected of causal mechanisms. One important property is transferability, motivated by the modularity and generalization ability of causal mechanisms.
-
Robot mechanism transfer (Fig. 5). Motivated by this property, we evaluate whether the learned action-conditioned dynamics transfer to previously unseen robots. Specifically, we train on Baxter, Kuka, and WidowX and evaluate on held-out Franka and Sawyer robots without adaptation. Despite substantial changes in robot morphology, appearance, and embodiment, A-JEPA maintains rollout above 0.90 at horizon 20, while the baselines exhibit considerably stronger degradation. These results provide indirect empirical evidence that A-JEPA learns more transferable causal dynamics, rather than merely capturing relationships specific to the training environments.
-
Visual intervention experiments (Appendix I). We also provide a qualitative evaluation of the learned causal mechanisms through latent interventions on Reacher. Specifically, we first identify the physical semantics of individual representation coordinates using MCC and examine the learned causal dependencies between them. We then intervene on individual coordinates and propagate their effects through the learned transition model. As shown in Fig. 11, intervening on the proximal-joint coordinate affects both proximal and distal joints, whereas intervening on the distal-joint coordinate produces a more localized effect. These observations are consistent with the learned causal direction and provide additional evidence that the learned representations capture meaningful causal dependencies.
-
OOD planning and counterfactual mechanism evaluation (new Appendix M). To further strengthen the empirical evidence, we have added two evaluations.
First, we evaluate in-distribution and OOD planning on the AnyGoal benchmark, covering four visual task families and their variants. A-JEPA achieves comparable in-distribution planning performance while outperforming LeWM on several OOD variants. For example, on Cube-Hop, A-JEPA achieves 61.3% success, compared with 27.6% for LeWM. These results provide further evidence of the generalization ability of the learned causal dynamics.
Second, we introduce a more direct counterfactual mechanism evaluation on OGBench Cube. We restore the same held-out physical state in MuJoCo and apply 27 different actions, allowing us to measure their actual effects on individual physical factors. Using only one matched latent coordinate per physical factor, A-JEPA achieves a mean direct-effect of 0.889, compared with -0.328 for LeWM. For contact-mediated cube effects, the corresponding scores are 0.568 and -0.721, respectively. Furthermore, A-JEPA achieves 79.85% success in counterfactual action selection, compared with 10.73% for LeWM, using the same physical-state planning cost and candidate actions.
We hope these clarifications and additional experiments address the reviewer's concerns regarding the practical benefits of causal representations and the evidence for causal mechanism recovery. We would appreciate it if the reviewer could take these new results into consideration when reassessing our work.
Please let us know if you have any further concerns or questions. We would be happy to provide additional clarification.
Thanks for the authors' response and clarification. My concerns regarding evidence of causal discovery in realistic environments are addressed due to counterfactual experiments. But I still have concerns about the motivation of modeling the underlying latent causal states. The authors claim that its motivation is for better generalization or task transferability, but there seems to be only one experiment in the main text supporting this claim. I think this claim is not well-justified and whether the performance gain comes from the causal state modeling is unclear. Hence, I will keep my current score.
Q1: Advantages of component-wise causal representations. Thanks for the comments. We would like to clarify that information preservation and component-wise causal recovery address different questions. An invertible representation can preserve all state information and support in-distribution prediction and planning. However, it does not necessarily recover individual causal variables or the corresponding causal mechanisms. We highlight three advantages of component-wise causal representations.
1. Causal interpretability. Component-wise identifiability establishes that individual learned coordinates correspond to latent causal variables up to permutation and component-wise invertible transformations. This provides a causal interpretation for individual representation coordinates. As shown in Fig. 4, A-JEPA consistently achieves higher MCC across four visual environments. Importantly, the baselines can preserve substantial state information without recovering individual causal factors. For example, on OGB, PLDM achieves a nonlinear state R2 of 0.989 but an MCC of only 0.485. This illustrates that information preservation alone does not guarantee component-wise causal recovery.
2. Intervention and counterfactual analysis. When individual representation coordinates correspond to causal variables, interventions can be performed on specific coordinates and their effects can be examined through the learned transition model. In Appendix I (Visual Intervention Experiments on Reacher), we demonstrate that intervening on the proximal-joint coordinate affects both proximal and distal joints, whereas intervening on the distal-joint coordinate produces a more localized effect. These results are consistent with the learned causal direction. Furthermore, in the newly added Appendix M, we directly evaluate counterfactual action effects on OGBench Cube. Using only one matched latent coordinate per physical factor, A-JEPA achieves a mean direct-effect R2 of 0.889, compared with -0.328 for LeWM. This provides additional evidence that A-JEPA learns action-conditioned mechanisms whose physical effects are accessible through individual representation coordinates.
3. Generalization through causal mechanism modularity. A key property of causal models is the modularity of their mechanisms: when one causal mechanism changes, other mechanisms need not change accordingly. Recovering individual causal variables provides an inductive bias toward learning such reusable mechanisms. In contrast, arbitrary invertible transformations can mix different causal variables, making the corresponding transition mechanisms more entangled and potentially sensitive to changes in the environment. Our Robot Mechanism Transfer experiments (Sec. 5.3) provide empirical support for this advantage. A-JEPA maintains rollout R2 above 0.90 at horizon 20 on previously unseen Franka and Sawyer robots, while the baselines exhibit substantially stronger degradation. We further added planning experiments on the AnyGoal benchmark (Appendix M). A-JEPA achieves comparable in-distribution planning performance but improves over LeWM on several OOD variants. For example, on Cube-Hop, A-JEPA achieves 61.3% planning success, compared with 27.6% for LeWM.