acceptodds
Under review as a conference paper at ICLR 2027

I Act Therefore I Am: When Is JEPA's Action-Conditioning Enough to Learn Causal Mechanisms?

Abstract

Recent empirical and theoretical advances suggest that joint-embedding predictive architectures (JEPAs) may learn promising representations for action-conditioned prediction of future representations. However, accurate prediction does not, in general, necessarily imply recovery of the underlying causal states that give rise to the observed dynamics. This work investigates when and how JEPAs can recover the underlying causal states from observations. We first introduce a latent variable model, in which high-dimensional observations are generated from latent causal states whose dynamics are governed by action-conditioned transition mechanisms. Based on this formulation, we develop a general information-theoretic objective that combines conditional likelihood maximization for learning transition dynamics with entropy maximization for preserving latent state information. We then establish identifiability conditions under which representations learned by this general objective recover the underlying latent causal states up to component-wise invertible transformations and permutation. One key condition for such identifiability is sufficient action-induced variation in the transition mechanisms. Guided by this finding, we instantiate the general objective with an action-modulated Gaussian additive-noise model, yielding action-modulated JEPA (A-JEPA). Experiments on synthetic environments verify the theoretical findings under the identifiability conditions and robustness to moderate violations, while visual benchmarks demonstrate improved state recovery and transfer to unseen transition mechanisms.

Primary area:
unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords:
Joint-embedding predictive architectures, Representation Learning

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_85QtOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies whether world models can learn representations that correspond to underlying causal variables. It formulates the problem using a dynamical system with latent causal states and proves that such representations can be learned under invertibility and other assumptions. Motivated by this, this paper proposed the method action-modulated JEPA (A-JEPA). Extensive experiments show the effectiveness of this method over multiple tasks.

Critical strengths
  1. The authors formulate a model to analyze latent causal dynamics and introduce assumptions like invertibility to characterize that the learned representations could match the underlying causal variables. Inspired by this theory, they propose an Action-Modulated additive-noise model and yield A-JEPA from this.
  2. The authors implement controlled experiments to show the proposed method's capabilities in latent-state recovery, latent-structure recovery, and visual dynamics prediction.
Critical weaknesses
  1. Causal representation learning is a important topic. However, its importance for dynamic prediction is unclear: The causal representations discussed in the paper may be equivalent, through an invertible transformation, to representations obtained through conventional training, without a loss of information. Therefore, the additional contribution of causal representations to our understanding of world models remains unclear.
  2. Beyond controlled environments, the experiments do not sufficiently show whether the model learns causal representations or why it learns these representations in real-world settings. Additional experiments in more realistic settings or further ablation studies would help clarify these points.
Questions for authors
  1. Although component-wise causal identifiability is theoretically stronger than information preservation up to an arbitrary invertible transformation, its practical benefits remain unclear to me. Could the authors clarify what additional advantages causal representations provide for world models compared with equally information-preserving representations?
  2. In Figure 5, the transfer performance is evaluated through latent rollout prediction. Could the authors clarify to what extent these results demonstrate the recovery of underlying causal mechanisms, rather than improved predictive representations? Would experiments involving controlled interventions provide more direct evidence?
3 comments

Q1: Advantages of component-wise causal representations.\color{red}{\text{Q1: Advantages of component-wise causal representations.}} Thanks for the comments. We would like to clarify that information preservation and component-wise causal recovery address different questions. An invertible representation can preserve all state information and support in-distribution prediction and planning. However, it does not necessarily recover individual causal variables or the corresponding causal mechanisms. We highlight three advantages of component-wise causal representations.

1. Causal interpretability. Component-wise identifiability establishes that individual learned coordinates correspond to latent causal variables up to permutation and component-wise invertible transformations. This provides a causal interpretation for individual representation coordinates. As shown in Fig. 4, A-JEPA consistently achieves higher MCC across four visual environments. Importantly, the baselines can preserve substantial state information without recovering individual causal factors. For example, on OGB, PLDM achieves a nonlinear state R2R^2 of 0.989 but an MCC of only 0.485. This illustrates that information preservation alone does not guarantee component-wise causal recovery.

2. Intervention and counterfactual analysis. When individual representation coordinates correspond to causal variables, interventions can be performed on specific coordinates and their effects can be examined through the learned transition model. In Appendix I (Visual Intervention Experiments on Reacher), we demonstrate that intervening on the proximal-joint coordinate affects both proximal and distal joints, whereas intervening on the distal-joint coordinate produces a more localized effect. These results are consistent with the learned causal direction. Furthermore, in the newly added Appendix M, we directly evaluate counterfactual action effects on OGBench Cube. Using only one matched latent coordinate per physical factor, A-JEPA achieves a mean direct-effect R2R^2 of 0.889, compared with -0.328 for LeWM. This provides additional evidence that A-JEPA learns action-conditioned mechanisms whose physical effects are accessible through individual representation coordinates.

3. Generalization through causal mechanism modularity. A key property of causal models is the modularity of their mechanisms: when one causal mechanism changes, other mechanisms need not change accordingly. Recovering individual causal variables provides an inductive bias toward learning such reusable mechanisms. In contrast, arbitrary invertible transformations can mix different causal variables, making the corresponding transition mechanisms more entangled and potentially sensitive to changes in the environment. Our Robot Mechanism Transfer experiments (Sec. 5.3) provide empirical support for this advantage. A-JEPA maintains rollout R2R^2 above 0.90 at horizon 20 on previously unseen Franka and Sawyer robots, while the baselines exhibit substantially stronger degradation. We further added planning experiments on the AnyGoal benchmark (Appendix M). A-JEPA achieves comparable in-distribution planning performance but improves over LeWM on several OOD variants. For example, on Cube-Hop, A-JEPA achieves 61.3% planning success, compared with 27.6% for LeWM.

Q2: Evidence of causal mechanism recovery in realistic environments.\color{red}{\text{Q2: Evidence of causal mechanism recovery in realistic environments.}} Thanks for raising this concern. We would like to clarify that, in real-world environments, the underlying causal variables and mechanisms are generally unknown, making their direct evaluation challenging. Similar to existing work in causal representation learning community, we therefore consider indirect evaluations that examine properties expected of causal mechanisms. One important property is transferability, motivated by the modularity and generalization ability of causal mechanisms.

  1. Robot mechanism transfer (Fig. 5). Motivated by this property, we evaluate whether the learned action-conditioned dynamics transfer to previously unseen robots. Specifically, we train on Baxter, Kuka, and WidowX and evaluate on held-out Franka and Sawyer robots without adaptation. Despite substantial changes in robot morphology, appearance, and embodiment, A-JEPA maintains rollout R2R^2 above 0.90 at horizon 20, while the baselines exhibit considerably stronger degradation. These results provide indirect empirical evidence that A-JEPA learns more transferable causal dynamics, rather than merely capturing relationships specific to the training environments.

  2. Visual intervention experiments (Appendix I). We also provide a qualitative evaluation of the learned causal mechanisms through latent interventions on Reacher. Specifically, we first identify the physical semantics of individual representation coordinates using MCC and examine the learned causal dependencies between them. We then intervene on individual coordinates and propagate their effects through the learned transition model. As shown in Fig. 11, intervening on the proximal-joint coordinate affects both proximal and distal joints, whereas intervening on the distal-joint coordinate produces a more localized effect. These observations are consistent with the learned causal direction and provide additional evidence that the learned representations capture meaningful causal dependencies.

  3. OOD planning and counterfactual mechanism evaluation (new Appendix M). To further strengthen the empirical evidence, we have added two evaluations.

First, we evaluate in-distribution and OOD planning on the AnyGoal benchmark, covering four visual task families and their variants. A-JEPA achieves comparable in-distribution planning performance while outperforming LeWM on several OOD variants. For example, on Cube-Hop, A-JEPA achieves 61.3% success, compared with 27.6% for LeWM. These results provide further evidence of the generalization ability of the learned causal dynamics.

Second, we introduce a more direct counterfactual mechanism evaluation on OGBench Cube. We restore the same held-out physical state in MuJoCo and apply 27 different actions, allowing us to measure their actual effects on individual physical factors. Using only one matched latent coordinate per physical factor, A-JEPA achieves a mean direct-effect R2R^2 of 0.889, compared with -0.328 for LeWM. For contact-mediated cube effects, the corresponding scores are 0.568 and -0.721, respectively. Furthermore, A-JEPA achieves 79.85% success in counterfactual action selection, compared with 10.73% for LeWM, using the same physical-state planning cost and candidate actions.


We hope these clarifications and additional experiments address the reviewer's concerns regarding the practical benefits of causal representations and the evidence for causal mechanism recovery. We would appreciate it if the reviewer could take these new results into consideration when reassessing our work.

Please let us know if you have any further concerns or questions. We would be happy to provide additional clarification.

Thanks for the authors' response and clarification. My concerns regarding evidence of causal discovery in realistic environments are addressed due to counterfactual experiments. But I still have concerns about the motivation of modeling the underlying latent causal states. The authors claim that its motivation is for better generalization or task transferability, but there seems to be only one experiment in the main text supporting this claim. I think this claim is not well-justified and whether the performance gain comes from the causal state modeling is unclear. Hence, I will keep my current score.

Reviewer_pa5LOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

This paper addresses an important question at the intersection of JEPA and causal representation learning, under what conditions can an action-conditioned predictive model recover the underlying causal states, rather than merely learn representation for predicting future observations? The paper establishes conditions under which the learned representations recover latent causal states up to permutation and component-wise invertible transformations. It further proposes A-JEPA based on action-dependent Gaussian transition mechanisms, and evaluates on synthetic and visual environments.

Critical strengths

S1. The research problem is important and well-motivated. The distinction between predictive representations and causal representations is fundamental to world models. It is important to know whether, when, and how conventional JEPA, which emphasizes prediction and representation quality, could ensure to derive causal factors.

S2. Recovering latent causal variables up to permutation and component-wise invertible transformations (under assumptions) is substantially stronger than recovering an arbitrary invertible transformation of the latent state. I would value this as an important theoretical contribution.

Critical weaknesses

W1. The paper emphasizes learning causal mechanisms (e.g., in the title), but its main theoretical guarantee concerns recovering the individual latent causal variables up to component-wise transformations. The causal mechanism includes causal graph structure and structural causal equations which are needed for interventional and counterfactual analysis. The paper needs to explicitly state whether and to what extent causal structure and structural causal equations could be discovered by or relied upon (see W2) in the proposed approach.

W2. The Gaussian transition model in Eq. (13) assumes that each learned component depends on a specified subset of parent variables, which indicates reliance on predefined causal structure in the practical model. As the causal structure is not discovered from data, this may raise concerns when comparing A-JEPA with baselines that do not receive equivalent structural information.

Questions for authors

The proof of Theorem 3.3 seems to rely on a factorization of the learned conditional transition density across representation coordinates. The authors need to clarify. If the factorization is required, this is an important assumption and should be explicitly listed along with Assumptions 3.1 and 3.2.

2 comments

Q1: Recovery of causal mechanisms.\color{red}{\text{Q1: Recovery of causal mechanisms.}} Thanks for the comment. We would like to clarify that this connection has been established in causal representation learning. Previous studies have shown that once latent causal variables are identified up to component-wise transformations, their causal structure can also be recovered under standard causal assumptions (Brehmer et al., Weakly Supervised Causal Representation Learning, NeurIPS 2022.; Liu et al., Identifying Weight-Variant Latent Causal Models, JMLR, 2026, Corollaries 2–3). We also empirically evaluate causal graph recovery in Fig. 2.

Q2: Predefined causal order.\color{red}{\text{Q2: Predefined causal order.}} We would like to clarify that the predefined causal order in the representation space is without loss of generality, as established in previous causal representation learning studies (Squires et al., Linear Causal Disentanglement via Interventions ICML 2023, Remark 1; Liu et al., Identifying Weight-Variant Latent Causal Models, JMLR 2026, Sec. 3.4). Specifically, due to permutation indeterminacy, the latent causal variables have no fixed ordering, and any permutation can be absorbed into the unknown observation mapping without changing the observed data distribution. Therefore, fixing a causal order in the learned representation space simply selects one representative among permutation-equivalent models. It does not require or use any ground-truth causal ordering or graph information. This is also explained in Appendix F.

Q3: Factorization of the learned transition model.\color{red}{\text{Q3: Factorization of the learned transition model.}} Thanks for pointing out this. We would like to clarify that the factorization of the learned transition model is motivated by our causal generative model. Specifically, the independent causal mechanism in the generative SCM imply that the conditional transition distribution factorizes as shown in Eq. (3). It is therefore natural to incorporate this factorization as prior knowledge when designing the learned transition model, ensuring consistency with the independent causal mechanisms assumed in the generative process. In other words, the independent causal mechanism in our causal generative model implies a factorized transition distribution, which naturally motivates imposing the same factorization on the learned transition model. This is analogous to adopting a Gaussian transition model when the underlying generative process is assumed to follow a Gaussian ANM.


We hope these clarifications adequately address the reviewer's concerns and clarify the theoretical foundations of our approach. We would appreciate it if the reviewer could take these points into consideration when reassessing our work.

I would like to thank authors for providing their clarifications in their rebuttal. Those clarifications can help readers understand better their theoretical contributions. I have increased my rating to weak accept.

Reviewer_6JhPOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
3
Soundness
1
Read review
Paper summary

The paper investigates the identifiability of the latent variables in JEPA architecture. Its learning objective combines next state prediction with maximizing entropy of the latent variables. It further identifies the assumptions under which this learning objective leads to the recovery of the true latent states up to component-wise transformations. Then, it proposes a practical implementation, A-JEPA, built on an action-modulated Gaussian additive-noise model in latent space. This model satisfies the variability assumption when the action modulates both the conditional mean and the variance diversely enough. The resulting method, A-JEPA, recovers state coordinates better than LeWM and PLDM on synthetic and visual benchmarks.

Critical strengths
  • The paper is written clearly and it is easy to follow.
  • The visual experiments show empirical improvements in latent variable estimation in terms of the MCC metric, compared to LeWM and PDML. The transfer experiment is in the right direction, yet it is not convincing enough to motivate the coordinate-wise recovery of the true latent states.
Critical weaknesses
  • The core identifiability result already exists. LEAP [Yao+21] already has the same variability assumption (Assumption 3.2) and the component-wise recovery argument (Theorem 3.3) for the same class of latent variable models, with actions replaced by regimes. This paper's potential contribution is connecting the existing theory to JEPA training without an observation decoder. However, the paper fails to cite this prior work in the main text or clearly distinguish its theoretical contribution. Extending existing identifiability results to JEPA without demonstrating clear practical benefits feels incremental.
  • The theory has an error: for λ>1\lambda > 1, the learned latents z\mathbf{z} are uniformly distributed over the unit cube and hence have independent components. This contradicts the core component-wise identifiability result, since the components of s\mathbf{s} are not marginally independent. This is stated as a result in Theorem 2.1 and as a condition in Theorem 3.3. I think the proof of Theorem 3.3 might still hold by setting λ=1\lambda = 1, but this further weakens the claim of proposing a general information-theoretic objective.
  • The paper does not motivate the use of a Gaussian transition model for latent variables z\mathbf{z} defined on the unit cube. This creates a significant gap between the theory and the practical implementation. Furthermore, the practical implementation assumes an action-modulated Gaussian additive-noise model in the latent space s\mathbf{s}. It is unclear how applicable this assumption is to complex real-world environments.
  • The synthetic experiment does not reproduce the setting the action-conditioned architecture is designed for. It holds the action constant over each 2000-step episode, so the action acts as a per-episode regime indicator, in the role of the auxiliary variable in LEAP [Yao+21]. In the control and planning settings that motivate the paper, actions are chosen at every step from the current state, not fixed for an entire episode.
  • The visual experiments show empirical improvements in latent variable estimation, measured by the MCC metric, compared to LeWM and PDML. However, they do not convincingly demonstrate how these improvements translate into better planning or transfer of planning capabilities to new environments. I would suggest an experimental setup that directly evaluates how learning causal processes affects in-domain and out-of-domain planning on established transfer benchmarks.
Questions for authors
  • Does Theorem 3.3 still follow with λ=1\lambda = 1? How does this choice affect the paper's claim of proposing a general information-theoretic objective?
  • Do the authors think that coordinate-wise recovery of the latent states is necessary for successful planning or transfer? What is their intuition on why the up-to-rotation recovery of the previous methods is not enough?
  • What are the established transfer benchmarks for planning suitable to JEPA architecture to test the hypothesis whether the coordinate-wise recovery of the latent states is necessary (vs. up-to-rotation recovery)?
3 comments

Q1: Relation to LEAP [Yao+21]\color{red}{\text{Q1: Relation to LEAP [Yao+21]}} Thanks for your comments. We take the LEAP very seriously. First, to highlight it, we discuss LEAP as the first representative work in the causal representation learning part of our Related Work section, alongside other studies that exploit similar variability conditions for identifiability. Second, we would normally include Related Work in the main text. However, due to the original nine-page limit, we placed it in the appendix, anticipating the additional page typically available at the camera-ready stage. Finally, to address the reviewer's concern, we have now added an explicit discussion of LEAP in the revised main text (end of Sec. 3).

Q2: Marginal independence v.s. unit-cube latents, and unit-cube latents v.s. Gaussian transitions.\color{red}{\text{Q2: Marginal independence v.s. unit-cube latents, and unit-cube latents v.s. Gaussian transitions.}} We sincerely thank the reviewer for carefully examining our theoretical results and, in particular, for suggesting possible fixes, such as setting λ=0\lambda=0. Motivated by this, we revisited our proof, and found that the learned representation does not need to be constrained to [0,1]d[0,1]^d. In fact, we only require its entropy to be upper-bounded by a constant kk. This removes the unintended marginal-independence constraint and resolves the inconsistency with component-wise identifiability, while retaining λ>1\lambda>1. Also, the Gaussian transition model is no longer required to be defined on the unit cube. We have revised the proof accordingly, please see the changes highlighted in red in Appendix C, where only the entropy-related part of the proof required modification.

Q3: Applicability of the Gaussian additive-noise model.\color{red}{\text{Q3: Applicability of the Gaussian additive-noise model.}} We would like to clarify three points. First, our theoretical results allow general nonparametric transition models and do not require Gaussian noise. Second, we consider the action-modulated Gaussian ANM for the reasons explained at the beginning of Sec. 4 (lines 228–238). We emphasize that this choice also illustrates how our general theory can be translated into a practical method. Third, our synthetic experiments explicitly examine violations of the Gaussian assumption.

Q4: Fixed actions in synthetic experiments.\color{red}{\text{Q4: Fixed actions in synthetic experiments.}} Thanks for such insightful comments. We would like to clarify the following points. First, our synthetic experiments are designed to verify the identifiability theory of the action-conditioned generative model, where the same action can be applied to different states. That said, Holding actions fixed within each episode is a controlled data-generation choice. Second, we agree that in control and planning, actions are typically selected based on the current state. However, the action-selection policy p(at∣st)p(a_t\mid s_t) is distinct from the transition mechanism p(st+1∣st,at)p(s_{t+1}\mid s_t,a_t) that A-JEPA aims to learn. Once learned, the action-conditioned transition model can support state-dependent, time-varying action selection, provided that the relevant state-action transitions are accurately modeled. Third, to directly evaluate this setting, we have added planning experiments on the AnyGoal benchmark, where actions are selected based on current observations, please see Appendix M.

Q5: Is component-wise recovery necessary for planning?\color{red}{\text{Q5: Is component-wise recovery necessary for planning?}} We do not claim that component-wise recovery is necessary for successful planning. For in-distribution planning, an accurate transition model can support planning even when the latent states are recovered up to an orthogonal transformation, since orthogonal transformations preserve Euclidean distances (as established by Klindt et al. (2026) for LeWM). However, as highlighted in our Introduction, one of the main motivations for learning causal states is to improve generalization under environmental changes. The key intuition is that causal models characterize the underlying transition dynamics through independent causal mechanisms, which can remain invariant and be reused under distribution shifts. In contrast, an orthogonal transformation can mix different causal variables within each representation coordinate. Consequently, individual causal mechanisms are no longer separated at the representation-coordinate level, making it more difficult to exploit their invariance when encountering previously unseen state-action environments.

Our existing Robot Mechanism Transfer experiments already provide empirical evidence supporting this intuition. To further evaluate its implications for planning, we have added AnyGoal experiments in Appendix M, where A-JEPA achieves comparable in-distribution planning performance while outperforming LeWM (Note: Klindt et al. (2026) establish identifiability only up to an orthogonal transformation for LeWM under certain conditions) on several OOD variants.

Q6: Additional planning and transfer evaluation.\color{red}{\text{Q6: Additional planning and transfer evaluation.}} We thank the reviewer for this suggestion. We have added extensive planning experiments on the AnyGoal benchmark (Appendix M, Fig. 12), covering four base tasks (TwoRoom, Cube, PushT, and InsertPeg) and their OOD variants. We compare A-JEPA with both an architecture-matched LeWM and Official LeWM, using the same CEM planner, start-goal pairs, and success criteria. All models remain frozen during OOD evaluation. The results show that A-JEPA achieves comparable in-distribution planning performance (e.g., 95.6% vs. 94.4% on TwoRoom and 28.0% vs. 29.1% on Cube), while outperforming LeWM on several OOD variants (e.g., 61.3% vs. 27.6% on Cube-Hop and 36.0% vs. 24.0% on AvoidPeg).

To further examine whether these improvements arise from learning individual causal mechanisms, we conduct counterfactual mechanism evaluations on OGBench Cube (Figs. 13–14). Using component-only readouts, A-JEPA achieves a direct action-effect R2R^2 of 0.889 versus -0.328 for LeWM, and 79.85% versus 10.73% success in counterfactual action selection.

Together, these results provide empirical evidence for the practical benefits of component-wise causal representations, particularly under distribution shifts. Please see Appendix M (Figs. 12–14) for the complete experimental setup, results, and analysis.


We sincerely thank the reviewer for these constructive comments. We hope that the above clarifications adequately address the concerns and will be taken into consideration when reassessing our work.

I thank the authors for their detailed comments. While some of my concerns are addressed, I cannot increase my score since the concerns I list below are not resolved convincingly.

  • [W1: LEAP] LEAP is cited in Related Work but is not discussed as the authors claim in their response. I appreciate the newly added discussion at the end of Section 3. The paper should further cite LEAP at the point where Assumption 3.2 is introduced, and revise the Introduction and contribution list to better distinguish the paper's contributions from this prior work. Nevertheless, this clarification weakens the novelty claim of the main theoretical contribution.
  • [W2: Error in theory] Changing the support of the latent variables is a substantial update to the theory, which requires a new round of review to understand how this change affects the rest of the assumptions/theory.
  • [W4: Mismatch between the synthetic experiment and Action-conditioned JEPA setups] The synthetic experiment (with a fixed action per 2000-step episode) does not reflect the real-world motivation or practical use cases of action-conditioned JEPA.
  • [W5: Experiments on planning and planning transfer] I appreciate the new planning and planning transfer experiments, which substantially address my original concern. However, I still think these should be presented as the main empirical results, with the toy synthetic experiment using fixed actions presented as a supplementary verification of the theory.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.