acceptodds
Under review as a conference paper at ICLR 2027

Do-JEPA: From Masking to Intervention in Latent World Models

85 views

Abstract

Latent world models are trained to predict what happens next, so nothing in their objective separates what an action caused from what merely co-occurred with it. Object-masking models such as C-JEPA intervene on what the predictor can see; we intervene on what physically happens. From one saved simulator state we run the dynamics under an action and under a reference action , and train the model to predict the difference between the two latent futures. The resulting objective, Do-JEPA, has an effect loss, a support loss (where the action enters), a propagation loss (where its effect travels) and invariance losses (what must not change). In a synthetic system with object-aligned variables, support supervision finds the directly intervened object in of test cases, where a sparse action mask sends the action to a nuisance slot in every case, and response-onset supervision recovers the ring-shaped propagation graph (edge AUROC vs. ). From pixels, the effect loss beats a control trained on exactly the same data: it lowers latent effect error by on an end-to-end LeWM model ( seeds) and physical effect error by when trained and tested on natural action sequences, and on three independently generated CausalWorld benchmarks it lowers responsive effect error by about under physics shifts and the latent context sensitivity of predicted effects by . Trained from scratch it costs factual accuracy; fine-tuning an existing model with it removes this cost. Together, these results show that intervening on the world, rather than on what the model sees, helps latent world models predict what their actions cause.

Primary area:
causal reasoning
Keywords:
world models, JEPA, paired interventions, action-effect learning, causal propagation, context invariance

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_PPK7OpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes Do-JEPA, a training framework that trains latent world models to identify the causal impacts of specific actions. From an initial simulated state, Do-JEPA generates paired trajectories respectively with the target and the reference actions. The models are trained to predict trajectory differences (i.e., impacts of the target action) as well as vanilla next-state predictions. Do-JEPA also proposes a series of sub-losses to provide fine-grained supervisions. Experimental results show that Do-JEPA manages to improve latent world models' capability to identify impacts of actions in different task scenarios.

Critical strengths
  • Clear method design: the method proposes to learn from paired trajectories with multiple losses to provide fine-grained supervisions.
  • Extensive experiments with strong results: the experiments include multiple environments, task settings, ablations, and random seeds, and demonstrate the performance of the method.
Critical weaknesses
  • Difficulty in data construction: the method requires contrastive control on environment states, which may be difficult to achieve in real-world scenarios.
  • Unclear practical significance: there is no concrete evidence that such learning can improve models' overall task completion performance. Specifically in planning, the models suffer from performance degradation.
Questions for authors
  • How can the method be applied to real-world scenarios where paired trajectories are difficult to construct?
  • Can the authors provide additional results or demos about joint validation of all modules?
  • Can the authors provide additional results to validate that the proposed training framework can improve models' overall task completion performance?
Reviewer_T4LUOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

DO-JEPA trains latent world models using two simulator rollouts from the same saved state, sharing context and noise but differing in action. It supplements branch prediction with a loss on the difference between their latent futures. Additional losses supervise action-entry support, response propagation, and invariance where suitable variables or context interventions are available. Experiments span synthetic systems, state-based Push-T, and pixel-based Push-T/CausalWorld. The strongest practical evidence concerns the paired-effect loss: physical effect prediction improves against a same-data control, particularly after fine-tuning, while the broader objective shows mixed results.

Critical strengths

The AUG control uses the same collected branches, helping separate the objective's benefit from additional data. Natural-action Push-T tests and three independently generated CausalWorld instances provide physical read-outs beyond latent error. The study is also informative about failure: it reports temporal aliasing, representation-dependent metrics, factual-accuracy costs, and unsuccessful confirmation criteria instead of presenting all variants as improvements.

Critical weaknesses

Evidence is stronger for PAIRED than for the proposed full framework. On frozen visual slots, full DO-JEPA increases physical effect error by 6.5% over AUG (Table 15); in CausalWorld, the added context term also hurts accuracy (Table 20). Gains depend on the intervention distribution (Table 18), and the planning study establishes compatibility rather than improved control (Appendix F).

The structural results need a cleaner ablation: Table 1 changes mask temperature alongside support supervision, so it does not isolate that loss. Propagation recovery is mainly demonstrated on synthetic rings; state-only gates also cannot exactly represent action-dependent response labels at a fixed state, as Appendix B acknowledges.

Novelty is positioned mainly against masking, while older counterfactual-physics methods are missing, including CoPhy (ICLR 2020) and Counterfactual Dynamics Forecasting (AAAI 2023). Their settings differ, but the distinction from prior interventional dynamics learning needs explanation.

Questions for authors
  1. Could Table 1 compare support supervision on/off at the same mask temperature and training settings, preferably over multiple seeds? How much of the target-localization gain remains?
  2. How should the state-only gates be interpreted when different actions at the same state yield different onset labels? Would conditioning gates on the action, or testing a branching interaction graph, change the propagation conclusions?
  3. Can the CausalWorld evaluation isolate a physics-only shift with the visual-context distribution fixed? The current mechanism split changes both factors. Paired uncertainty estimates for the physical gains would also help assess their reliability.
  4. How does the paired latent-difference objective differ from earlier counterfactual-physics and factual/counterfactual-difference modeling? Please distinguish the objectives, supervision, and test-time information required.
Reviewer_wkaiOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

Authors introduce DO-JEPA, a training framework for latent world models using paired interventions. It runs the simulator twice from the same state, using different actions, and trains the model to predict the difference between the outcomes. Experiments on synthetic systems, Push-T, and CausalWorld show improvements in action-effect prediction and, in controlled settings, intervention target and propagation graph recovery.

Critical strengths

The paper asks a meaningful question: does predicting the future well mean that a model understands what an action causes? The comparison using the same training data is useful, and I appreciate that the authors report and investigate the negative results.

Critical weaknesses

The design makes sense, but I am not yet convinced that the gains establish a sufficiently useful capability. The strongest structural results come from controlled settings, while the pixel-based experiments mainly show better effect prediction. These gains depend on the training setup and do not improve planning. The authors investigate these issues, but the overall benefit remains unclear to me.

Questions for authors
  1. Does better effect prediction help the model choose better actions? The planning results do not show a clear gain. Could you use the existing results to explain where the improved predictions help and why they do not translate into better planning?

  2. For pixel-based tasks, what does the full DO-JEPA objective add over the paired effect loss alone? The context loss seems to hurt some results. Which losses would you recommend using, and what evidence supports that choice?

Comments on the paper

2 comments

Dear authors and reviewers,

The author-reviewer discussion period closes on November 18 (AoE). Please engage in response periods so that we have our issues discussed thoroughly.

AC

Reviewer_wkaiOpenReview ↗

Final personal assessment

I didn't use AI to evaluate the paper. The review reflects my own assessment.

And, I am happy to clarify my reasoning with the AC if needed. But I dont want to repeat everything from my original review.

  1. why I initially gave a weak reject?

My main concern came from the controlled settings. The effect RMSE improved, but planning did not show a clear improve. In the original paper, not including appx, author didn't decline the plan success rate. Only in discussion and limitation say " with success equivalent to the base planner within ±3 percentage points." Personally, if it does not improve planning, this paper is less convincing to me.

  1. Why I changed my score?

You can see Q1 and W1, they already addressed my question, thank you!

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.