acceptodds
Withdrawn from ICLR 2027

Do-Causal-JEPA: Learning Interventional World Models with Graph Surgery Estimator

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Recent advances on Joint Predictive Embedding Architecture (JEPA) have shown significant progress in learning object-centric representations and the dynamics of these representations from images and videos. One prominent model is known as Causal-JEPA (C-JEPA) nam2026causaljepalearningworldmodels, which learns the interaction dynamics of objects by masking different feature maps based on images from previous time frames to predict the next time frame's representation state. The model is shown to be able to learn an important notion called influence neighborhoods, which are claimed to be the minimal predictor sets of each object in the image without causal sufficiency under mild assumptions. However, the relationship between influence neighborhoods and some classical causality concepts, such as invariant causal predictors, is not well-understood. In this work, we show that C-JEPA does not learn invariant causal predictors in a controlled environment in the presence of latent confounders. We propose an alternative training scheme to introduce an additional loss term that is based on graph surgery operations for learning object-level invariant predictors given a causal structure. Empirically, our proposed algorithm Do-CJEPA outperforms C-JEPA in learning the dynamics of the objects under mechanism shifts by having a smaller out-of-distribution test error gap and prediction errors in terms of mean squared error for both one-step prediction and rollout under shifted test environments.

Primary area:
unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords:
invariant prediction, world models, causality

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_LFJTOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This work discusses Do‑CJEPA, a modification of Causal‑JEPA (C‑JEPA) for object‑centric world modeling under distribution shift. Given frozen SAVi slots matched to object roles, it (i) restricts transformer attention for each target slot ZtrZ^r_t to its lag‑one parents in a slot‑level ADMG (or to possible parents of a PAG estimated with G‑CaRL + LPCMCI when no graph is given), and (ii) adds a stop‑gradient regression loss Ldo\mathcal{L}_{do} toward the mean of the graph‑surgery estimand P(Y∣do(IY))P(Y\mid do(I_Y)), IY=pa(MACS(Y))∖MACS(Y)I_Y = pa(\mathrm{MACS}(Y))\setminus \mathrm{MACS}(Y), estimated by ID‑GEN from source data only. The claim is that this yields predictors invariant to mechanism shifts in the presence of latent confounders, without knowing the shift target or time.

Critical strengths
  1. The evaluation protocol is structurally leak‑free and the intervention‑aligned analysis is a reusable contribution in its own right. The predictor does not receive t⋆t^\star, the intervened role, or the intervention value. ID‑GEN, the causal‑discovery pipeline, graph aggregation, and checkpoint selection all use source trajectories only. The k=t−t⋆k=t-t^\star alignment in 4.2, with the directly intervened token removed only at k=0k=0, is the right way to read a one‑time intervention: it separates the unavoidable error at the shifted node from error that propagates to nodes whose mechanisms are unchanged, which is the only quantity an invariance method can legitimately be judged on.

  2. The paper draws the correct conceptual line between predictive sufficiency and invariance, and respects it in the PAG variant. 2.2 correctly states that an influence neighborhood is a Bayes‑optimality object, not a causal one: a token can be in Nt(i)\mathcal{N}_t(i) precisely because it is a child of the selection variable. This is a sharper reading of C‑JEPA than C‑JEPA's own authors give. The same discipline shows up in 3 and footnote 4: a PAG is read only as a possible‑parent restriction (Y→XY\to X and Y↔XY\leftrightarrow X excluded), never promoted to an ADMG from which an identification functional would be computed.

Critical weaknesses
  1. By the paper's own Eq. 6, the interventional estimand equals the conditional, so the "do" contributes nothing, and the mechanism the main text credits for the gain is contradicted by the method as described. In Sec. 4 the authors write that in Interventional Pong MACS(Y)={Y}\mathrm{MACS}(Y)=\{Y\}, hence P(Yt∣do(IY))=P(Yt∣Zt−1B,Zt−1L,Zt−1R)P(Y_t\mid do(I_Y)) = P(Y_t\mid Z^B_{t-1},Z^L_{t-1},Z^R_{t-1}). This means (a) Ldo\mathcal L_{\rm do} regresses the predictor toward the same conditional mean that LJEPA\mathcal L_{\rm JEPA} already converges to under the same input restriction, i.e. it is self‑distillation from an 8‑component GMM; (b) the ID/IDC machinery of Sec. 2.1, Fig. 2 and the conditioning set WW of Def. 2.1 are never exercised; and (c) the "known ADMG" gives every target all three lag‑one slots, so the causal restriction is exactly "attend only to frame t−1t-1" (the Do‑JEPA bullet states each batch observes only frame t−1t-1). Nothing object‑specific is removed. Sec. 4.1 then explains the improvement as "target‑specific attention prevents invariant predictions from depending on the intervened right‑paddle slot," but the mask includes Zt−1RZ^R_{t-1} for every target; what it removes is older frames.

  2. The headline gains come mostly from how the baseline was trained, and the paper's own figures show this. C‑JEPA is trained for a single task, while the Do‑variants are trained to predict frame tt from the preceding frames for every t∈{2,…,15}t\in\{2,\dots,15\}, and all models are then evaluated at every tt. The baseline is tested on prediction lengths it ddid not see, and Fig. 4 shows the cost: on frames before the intervention (k<0k<0), where nothing has shifted yet, C‑JEPA already sits near 0.0100.010 while the Do‑variants sit near 0.0010.001, a gap on unshifted data that equals C‑JEPA's reported OOD gap in Fig. 3. So the abstract's 69%69\%/90%90\% reductions and "90%90\% smaller OOD gap" are largely measuring the baseline's handicap. What remains is further weakened by two facts: Sec. 4 describes the shift as a one‑time nudge to the right paddle after which the ordinary dynamics resume, to which any correct one‑step Markov model is invariant by construction with no causal knowledge; and in Fig. 5 the same‑slot OC‑DINO‑WM curve crosses below every Do‑variant and finishes lowest at t=15t=15.

Questions for authors
  1. Could you report matched‑boundary C‑JEPA with a context window of exactly 1 frame (and 2 frames), without any graph. If this reproduces the mask‑only row, what remains of the causal contribution in the ADMG variant?

  2. When is the do‑target different from the conditional? Eq. 6 gives P(Yt∣do(IY))=P(Yt∣IY)P(Y_t\mid do(I_Y)) = P(Y_t\mid I_Y) in the benchmark, and Sec. 3 falls back to P(Y∣IY)P(Y\mid I_Y) whenever YY is confounded with a lag‑one predictor. Could you please give one slot‑level graph in which the target is both identifiable and ≠P(Y∣IY)\neq P(Y\mid I_Y), and (b) evaluate Ldo\mathcal L_{\rm do} on a setting that instantiates it.

  3. Is there a hidden confounder? Could you report a linear probe from a single matched SAVi slot to simulator ball velocity? If it is accurate, Fig. 1b's bidirected edges are not the correct latent projection, and the success of a one‑frame‑context predictor is explained.

Reviewer_JsYyOpenReview ↗
Rating
1
Confidence
2
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

The paper aims at improving over the C-JEPA architecture by enabling the learning of object-slot-level invariant predictors in the presence of latent confounders. To this end, the Authors introduce an additional “Do” term in the loss, which is based on graph surgery estimators for learning object-slot-level invariant predictors given an acyclic directed mixed graph (ADMG). The proposed method is compared against the baselines on the Interventional Pong dataset (Lippe et al., 2022).

Critical strengths

I was not familiar with C-JEPA, as my expertise lies outside JEPA-style architectures. After reading the C-JEPA paper, I found the research question addressed by the Authors both important and interesting. I also believe the topic may be of broad interest for the applied ML community.

Critical weaknesses

The ADMG in fig.1b, given as input to the proposed Do-CJEPA method and its tested variants, is not compatible with the ground-truth data-generating causal structure (line 298) in fig.18 in Lippe et al. (2022); nor even with the “modified” version shown in fig.1a of the paper (see the details below). This is not just a typo in the graph. It has substantive negative consequences for the empirical assessment of Do-CJEPA in Sec. 4, since it impairs the key graph-surgery component introduced to replace and address the limitations of the conditional prediction targets of C-JEPA. This issue drives my recommendation to reject the paper: in its current form, the claims are not supported by the empirical evidence and fixing it would require substantial revision of the submitted material.

The ablation study in Table 1 (App. H) is consistent with this concern. The improvement over C-JEPA comes entirely from the layerwise lag-one masking; the “Do” component is irrelevant: the performance of “Full Do-JEPA (ADMG)”, which leverages Eq.(6), is statistically indistinguishable from “Mask only”. Moreover, since Eq. (6) assigns all three lag-one slots as parents, in reality “Mask only” encodes no graph-specific causal knowledge. It is equivalent to a first-order Markov truncation. The conclusion drawn by the Authors in lines 430-431 (“This shows the benefits of leveraging causal knowledge of the underlying mechanism of the observed data.”) is therefore not supported; those results only suggest that the observed gain is attributable to lag-one truncation under an incorrect input graph. Below are the reasons why the ADMG in fig.1b is problematic.

Fig.1b has three main issues. First, the directed edges Zt−1R→ZtLZ_{t-1}^R \to Z_{t}^L and Zt−1L→ZtRZ_{t-1}^L \to Z_{t}^R (that is, lagged causal relations between left and right paddles) do not follow from fig.18 of Lippe et al.(2022). Moreover, together with Zt−1B→ZtLZ_{t-1}^B \to Z_{t}^L, they are not even consistent with fig.1a in the paper, where the left paddle depends only on its own past and on the ball velocity direction and the right one depends on its own past, the ball x and y positions, and the ball velocity direction.

Second, fig.18 of Lippe et al. (2022) contains the causal relations ballt−1x→ball-vel-dirt\text{ball}^x_{t-1} \to \text{ball-vel-dir}_{t} and $ \text{ball}^y_{t-1} \to \text{ball-vel-dir}_t$. When the ball velocity chain becomes latent, the latent projection onto the slots therefore contains the directed edges ZtB→Zt+krZ_t^B \to Z_{t+k}^r , r∈{B,L,R}r \in \lbrace B,L,R\rbrace and k>1k>1, which are absent in fig.1b.

Third, the bidirected edges between the slot-level causal variables across time are missing. In more detail, since ball-vel-dir\text{ball-vel-dir} has a causal effect not only on the ball positions and paddles but also an autoregressive component ball-vel-dirt−1→ball-vel-dirt→ball-vel-dirt+1→…\text{ball-vel-dir}_{t-1} \to \text{ball-vel-dir}_{t} \to \text{ball-vel-dir}_{t+1} \to \ldots, its projection induces bidirected edges between all slot variables at all time steps. Formally, the set of bidirectional edges is
{Zti↔Zsj , with (i,t)≠(j,s) ; (i,j)∈{B,L,R}×{B,L,R}; t,s in the observation window} .\lbrace Z_t^i \leftrightarrow Z_s^j \ , \text{ with } (i,t)\neq (j,s) \ ; \ (i,j) \in \lbrace B,L,R\rbrace \times \lbrace B,L,R\rbrace ; \ t, s \text{ in the observation window}\rbrace \ . To allow the Authors to reproduce my derivation: I applied the two-step protocol in Prop. 3.6 of Gerhardus et al. (2023) to compute the finite marginal ts-ADMG of the infinite ts-DAG in fig.18 in Lippe et al. (2022), treating ball-vel-dir\text{ball-vel-dir} and score\text{score} as unobservable time series. Basically, with reference to Gerhardus et al. (2023), in the first step the Authors have to apply Props. 3.7 and 3.8 to marginalize over the set of vertices outside the observational window. Next, in the second step they compute the final finite marginal ts-ADMG by simply applying Def. 2.1. Finally, simply group the variables in slot variables (obviously this step is not in Gerhardus et al., 2023). To the best of my knowledge, the resulting finite marginal ts-ADMG not only invalidates Eq.(6), but also makes the interventional query non-identifiable from the observational distribution given the correct ts-ADMG.

Ref: Gerhardus, A., Wahl, J., Faltenbacher, S., Ninad, U., & Runge, J. (2023). Projecting infinite time series graphs to finite marginal graphs using number theory. arXiv preprint arXiv:2310.05526.

Questions for authors

I do not have any such questions, but I am still happy to discuss my major concerns with the Authors.

Reviewer_fxGyOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper proposes Do-CJEPA, a modification of C-JEPA that introduces causal structure to a JEPA model. Specifically, the authors note that C-JEPA is in fact an object-centric masked latent world model that does not guarantee causal invariance. The example used in the paper is Pong that has three objects (ball, left and right paddles), with a selection variable attached to the right paddle and hidden confounder based on the previous ball velocity direction. To address this issue, the proposed Do-CJEPA introduces some changes over C-JEPA: First, slots are aligned to the nodes of an object-level ADMG or PAG. A per-target graph suregery query is formed and an identifying functional of the target distribution under intervention is estimated by a conditional generator. Finally, transformer self-attention is restricted to the target's parents in the preceding frame. Validation experiments use a Pong variant with paired source / intervention trajectories.

Critical strengths

The motivation and idea are a good example of causal representation learning strategies. The implementation ideas are also valuable, showing how one can restrict a world model's predictor inputs to the preceding frame's object-level causal parents to improve robustness in interventional settings. In the matched factorial, the restriction accounts for a 52.5% reduction of shifted invariant MSE and 70.4% reduction of post-shift one-step error, although errors are quite small in terms of absolute value. The missing parent controls show the effect is specific to the graph restriction.

Honest and thorough ablations provided, along with fully artifacts with all the experimental details required, fully supporting reproducibility and future research.

Critical weaknesses

The results provided in Appendix H and Table 1 do not seem to support many of the claims made in the paper. For example, the matched baseline (which aligns the experiment parameters) at t = 15 rollout C-JEPA achieves the best results. Additional details and information about the limitations of the proposed method are needed.

Quantified results for the PHYRE evaluation are missing.

The causal-invariance contribution, as formalised, is not propertly validated. Table 1 and the λ\lambda sweep show that Ldo\mathcal{L}_{do} contributes nothing once the mask is present. The contribution of the graph surgery estimator is therefore not supported empirically.

Questions for authors
  1. Given Table 1 the flat λ\lambda sweep, and λdo=0\lambda_{do} = 0 for Do-CJEPA (PAG), what evidence supports the graph-surgery
    target as a contribution? Have you run any setting where the interventional target is necessary?

  2. Eq. 2–3 rely on MY=MACS(Y)M_Y = MACS(Y) and an ID functional, but the released known graph is dense and MACS(Y)=YMACS(Y) = {Y} for every target, so P(Y∣do(IY))=P(Y∣IY)P(Y | do(I_Y)) = P(Y | I_Y). How is Fig. 1b's confounded structure used in the reported runs, and does any reported query exercise the identification rules (Algorithms 2–3) rather than reducing to parent conditioning?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.