acceptodds
Under review as a conference paper at ICLR 2027

Learning to Ignore What Matters: JEPAs in Stochastic Worlds

Abstract

Joint-embedding predictive architectures (JEPAs) learn world models by predicting future representations rather than reconstructing observations. We show that this objective does not protect the information needed for control. We identify two failure mechanisms for jointly trained JEPAs in stochastic environments: the representation can discard the state that decisions depend on, and the predictor can lose decision-relevant features even when the representation keeps them. We test both by training agents entirely inside a JEPA world model and inside a generative world model with the same data and compute, then evaluating them in the real environment. The JEPA supports far weaker control: agents trained inside it survive up to 50 times less often and fail to learn multi-step behavior in Craftax. These failures are not caused by collapse, and rollouts that look correct can still teach agents to exploit errors in predicted rewards and terminations. Low latent prediction error and non-collapsed representations coexist with poor closed-loop performance, so neither is sufficient for control.

Primary area:
unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords:
world models, joint-embedding predictive architectures, JEPA, self-supervised learning, stochastic dynamics, model-based reinforcement learning, planning

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_r2wxOpenReview ↗
Rating
2(+1)
Confidence
3
Informativeness
1
Presentation
2(+1)
Soundness
2(+1)
Read review
Paper summary

The work explores two reasons why joint-embedding predictive architectures (JEPAs) fail in stochastic environments. The first is that the objective does not incentivize the encoder to keep information relevant to decision making, and the second is that chaining multiple one-step predictions of the next state does not produce the expected future state. The authors propose an alternative method that trains an encoder with a reconstruction objective and trains predictions of the next latent state with flow matching rather than regression. This method is compared to two JEPA variants in Craftax and a set of custom visual control environments. The latter environment offers several variants with differing types of stochasticity. The results show that the proposed model outperforms the JEPA variants, with the biggest difference occurring in most stochastic configurations.

Critical strengths

1. The custom environment with its stochastic variants is appropriate given the phenomenon the paper aims to test. The simplicity aids in interpreting the results.

Critical weaknesses

1. It is not clear what research question, if any, is being answered. The two mechanisms presented in Section 2 are largely known. The problem of the objective not incentivizing the encoder to keep information relevant to decision making is clear from the objective and does not require theory or experiments to verify. The issue with expectation models is also well understood both in work cited by the paper (Wan et al., 2019; Grimm et al., 2021), and by other work like that by Antonoglou et al. (2022) on stochastic MuZero. Diagnostic papers can be a great service to the community, but only when it is clear what they aim to diagnose. The work presents many experimental results and is densely packed with interpretations of the various pieces of each experiment. However, without knowing what question the authors’ aim to answer, I fail to find the purpose of the experiments and significance of the findings is unclear to me.

2. Experiments lack rigor and contain confounders. The most egregious example is that the flow matching method backpropagates gradients from the reward head through the next state prediction module, while both JEPA variants do not. This gives the flow matching method a reason to maintain information relevant to decision making that the JEPA models lack. This difference alone could account for why the flow matching model outperforms both JEPA variants, invalidating several of the other explanations proposed to explain the difference in performance.

The above two are my primary critiques, but the manuscript contains many more issues. The writing is hard to follow and often using pronouns with unclear antecedents, obscuring conclusions. Details essential for interpreting experiments (like how the readout in Test 1 is performed) are all relegated to the Appendix. The work draws strong conclusions from three seeds.

Questions for authors

Is there a central research question that unites the findings as part of a shared story, and that survives the confounding factor I mention in weakness 2?

2 comments
AuthorsOpenReview ↗

Response to Reviewer r2wx

You asked whether one research question unites the findings and whether it survives the reward-head confound. The submitted version did not state the question clearly or isolate the effect of task-head supervision. The revision addresses both concerns.

The question. Can an agent learn, inside a JEPA trained end to end, to handle an event that it cannot predict? If it cannot, is the fault in what the encoder keeps or in what the predictor forecasts? The revision is organized around this: the test (Section 2), the encoder (Section 3), the predictor (Section 4) and the repair (Section 5).

We agree that both mechanisms are known in principle. That the objective does not ask for decision-relevant state can be read off the objective, and expectation models are well understood. The related work now says this and cites Antonoglou et al. (2022) with Wan et al. and Grimm et al. What cannot be read off the objective is how a trained model behaves, and that is what the paper measures. The LeWorldModel encoder does see the wall, and the gradient through the prediction target shrinks the wall's weight in the latent by more than a thousand times. Stopping that gradient restores it. With background noise the encoder outputs noise at a variance that we predict without a free parameter. Inverse dynamics, task heads, a teacher and a stopped target all leave the agent unable to survive the wall, and on the fixed schedule only reversing both the learned target and the regression predictor fixes it. The failed models pass a collapse check and reach a low prediction loss, so we propose the agent test as the diagnostic.

The confound. The flow model's reward and termination heads did send gradient into its dynamics and LeWorldModel's did not, and that could have explained the gap. We tested it from both sides. A JEPA trained jointly with the same heads keeps the wall in its encoder (obstacle weight near 0.3, against 0.0001 without them), and its agents still survive 1% on the fixed schedule and no more than 5% in any obstacle world. A flow model whose heads send no gradient to the dynamics still survives 88% on the fixed schedule and 70% with random arrivals (Table 1).

The head gradient does matter in one place. With background noise or a background texture the flow model fails without it, and the paper states this in its limitations. We read it as support for the conclusion that relevance has to come from the task. The same supervision does not rescue the JEPA. With a stopped target, inverse dynamics and those heads, a policy trained on real transitions through the JEPA's encoder survives 73 to 99% in the four obstacle worlds, and agents trained inside the model stay below 9%. What is left is the regression predictor.

Writing, readouts, seeds. The main text was rewritten as one argument, and each readout is defined where it is first used (Sections 3.1 and 3.2). Table 1 reports means and 95% intervals over independently fitted world models. With three fits the intervals are wide and the caption says so. Despite the wide intervals, the separation in mean control performance is large: in the obstacle worlds all 81 JEPA fits remain below 9%, while the reconstruction model's means range from 72% to 94%. Section 4 no longer says that agents exploit the model. It reports what we measured: the regression models predict 14% deaths for their own policies and 44% occur, where the flow-matching models predict 14% and 16% occur.

Thank you for clarifying the research question and running additional experiments to address the confounder I raised. The revision is an improvement, and I have updated my presentation, soundness, and overall scores from 1 to 2. However, I still have substantial concerns.

Confound: I appreciate the new controls, but the comparison between a JEPA without task-head gradients and a flow model with them remains a headline result. The revision shows that task-head gradients largely prevent the JEPA encoder from suppressing task-relevant information, yet this suppression remains a central part of the paper's argument.

Motivation and significance: I now understand the research question, but I remain unclear about its significance. Once task supervision addresses the representation issue, the main remaining failure appears to be in the transition predictor. We already know that expectation models cannot accurately model multiple possible outcomes, and that generative models can. While the paper provides additional analysis of these failures, their significance is unclear. With the question of retaining task-relevant information largely resolved by reward head gradients, it is unclear why JEPA is central to the paper at all. The main finding seems to concern the known limitations of expectation models, rather than anything specific to JEPAs.

It has not affected my score, but I also find the authors' excessive use of AI, and the resulting convoluted prose, very frustrating. I have no objection to using AI, but it is the authors' responsibility to ensure that the final text is clear and accessible to human readers.

Reviewer_PzE5OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper examines representation and prediction failures in jointly trained JEPA world models. Controlled visual tasks and Craftax-Classic combine state readouts with policies trained in imagination. A generative reference, real-transition interface controls, oracle-mean decoding, and predictor swaps investigate how useful state information and low latent prediction error can coexist with poor control.

Critical strengths
  1. On the black gate, real-transition training through the frozen JEPA encoder–decoder achieves approximately 97% survival, versus 2% after imagined training. This shows that an unusable observation interface alone cannot explain the behavioral gap.

  2. Oracle-mean decoding shows that averaging the possible future encodings largely preserves the wall, while the learned predictor removes it. Predictor swaps and interventions on randomness, budgets, target gradients, and regularizer weights further constrain explanations. Reported negative results appropriately limit claims about possible repairs.

Critical weaknesses
  1. Predictable-nuisance shortcuts and prediction–control objective mismatch are established concerns acknowledged in Section 6, including Sobal et al. (2022), Lambert et al. (2020), and LeWorldModel's probing/planning results. The oracle-mean and swap controls refine particular diagnoses, and the background reversal adds an empirical observation. However, Appendix J finds a state-preserving wind fit with a lower original objective value, leaving the optimization mechanism behind the failure and reversal unexplained. The advance beyond a detailed case study is therefore not sufficiently established.

  2. The recipes differ in predictor design, history, training stages, and task supervision; the one-directional swap does not establish behavioral attribution. Both JEPA encoder designs rely substantially on Transformers, although the grid variant incorporates convolutions, so robustness across encoder families remains untested. Several behavioral comparisons use one model fit with three policy seeds, while noisy gates use independent fits. Craftax uses one model per family and has an acknowledged decoded-to-native interface mismatch (Appendices F, H, K, and O). These limits leave objective, architecture, optimization, and interface effects incompletely separated.

Questions for authors
  1. Which reported finding provides the main new mechanistic or practically useful insight beyond the nuisance-shortcut and objective-mismatch results discussed in Section 6? Which existing comparison best demonstrates that advance?

  2. In the existing termination probes (Figure 3; Appendix H), what does the comparison between real-encoded and predicted latents establish about head miscalibration versus dynamics-induced distribution shift?

2 comments
AuthorsOpenReview ↗

Response to Reviewer PzE5

The main new insight, and the comparison that shows it. Prior work says that a predictive objective need not keep what control needs. Our finding is more specific: the gradient through the learned prediction target removes a feature that the encoder can see. On the fixed schedule with a plain background the wall is on screen and nothing distracts from it. When the wall is removed from a frame, the squared change of the LeWorldModel latent, divided by the total latent variance, is still only 0.0001. When we stop the target gradient and change nothing else, that number is 0.40. LeWorldModel's stated advance is that SIGReg makes the stopped gradient unnecessary. SIGReg does prevent collapse, and the stopped gradient was also keeping the target from shrinking what is hard to predict. The comparison that best shows this is the two-by-two of Figure 4: 3% survival for LeWorldModel, 1% with the target gradient stopped, 2% with a flow-matching predictor, 98% with both.

Two further results are new as far as we know. With fresh background noise the encoder outputs noise, where Sobal et al. (2022) found that a JEPA ignores such noise. Equation 1 predicts the variance of that noise with no free parameter: 0.734 predicted, 0.733 to 0.734 in the three new fits, and within 0.7% on 23 earlier failed fits. And with the representation held fixed, regression fails as arrival times become random while flow matching does not: 17% against 91% mean survival with a geometric delay (Figure 3).

The state-preserving fit with a lower objective. You are right that this left the failure unexplained. The noise solution is far from the optimum: the objective is about 0.94 there and 0.23 for a fit that keeps the agent (Appendix C.4). Proposition 3 shows why training can stay there. Near the noise solution a weak copy of the state improves the prediction loss only at fourth order in its amplitude, so the gradient and the curvature both vanish. This is a calculation in a reduced model and does not describe the dynamics of the full network. Empirically the learning rate decides. At LeWorldModel's rate training ends on the noise solution. At a three times lower rate it leaves, and agents survive hidden drift in 79% of episodes. The wall is still lost at the lower rate, so the obstacle failure is a separate effect.

Recipe differences, encoder family, fits, Craftax. The squared-loss predictor of Table 1 now shares the flow model's tokenizer, network, optimizer, update count and task heads, and Appendix B.4 lists what still differs. We added behavioral evaluations of the swap and a separately trained squared-loss control on the same frozen stopped-target encoders: 10% survival against 97% with flow matching. A convolutional encoder lands on the same noise variance (0.735). It keeps more of the wall on the plain background than the transformer encoder does, and Appendix C.4 says so. Craftax now has three fitted models per family and one planner that searches in latent space, which removes the change from decoded frames to game pixels. The planner obtains a wooden pickaxe in 68% of episodes with the flow model and 1% with the JEPA, where random play obtains one in 0.4%. The probes that score the plans are fitted per model and their accuracy may differ (Appendix C.12).

Termination probes. On real encodings the head is close to the recorded termination rate: 0.048 against 0.053 at step eight under hidden drift. On the model's own predictions it outputs 0.91. The predicted latents are far from the head's training inputs, with norm 0.06 against 19.4 for real encodings. The large error appears on predicted inputs, consistent with a dynamics-induced distribution shift. The comparison cannot separate the transition error from the head's behavior on inputs it never saw, and the probes use random actions (Appendix C.11).

Thank you for the clarifications and additional experiments. The behavioral evaluations of the predictor swap, the comparison with the target gradient stopped, and the Craftax planner that searches in latent space strengthen the empirical evidence and address several of my concerns. However, the connection between removing a feature that the encoder can see and poor control remains only partially established, particularly since the substantial improvement in survival occurs only when stopping the target gradient is combined with a flow-matching predictor in the reported comparison. The calculation in a reduced model and the learning-rate experiments provide useful insight but do not fully explain the dynamics of the full network. As acknowledged, the termination probes cannot separate the transition error from the head’s behavior on inputs it never saw. Overall, I appreciate these improvements but retain some concerns about the strength and scope of the mechanistic conclusions.

Reviewer_2FKkOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

JEPA world models learn by predicting the latent of the next observation, and their objective rewards whatever is predictable. The paper argues that this is at odds with control in stochastic worlds, where the part of the state an agent controls is often the least predictable, so a JEPA can learn to ignore what matters for decisions (Section 1). It identifies two ways this happens (Section 2).

Mechanism 1, the representation drops the state: Nothing in the objective asks the encoder to keep the controlled state, and random perturbations of that state cost prediction loss. The encoder can therefore leave it out while staying non-collapsed, so the information is lost before any prediction is made (Proposition 1).

Mechanism 2, the rollout loses what the representation kept: A squared loss trains the predictor toward the conditional mean of the next latent. Means can mislead a planner and do not compose over a rollout, and features that barely affect the loss are the first to be lost to approximation error (Equation 2, Propositions 2-3). A sampling predictor can repair this mechanism, but not the first: if the encoder has dropped the state, no predictor can recover it.

Experiments: The paper compares the LeWorldModel JEPA with a flow-matching generative model in two settings: small visual control worlds where sources of randomness can be switched on one at a time, and Craftax, a larger open-world survival game. Test 1 probes what the latent and its rollouts keep, and shows both failures: the JEPA's encoder loses the controlled state in stochastic settings, and in one world its predictor deletes an obstacle the encoder had kept, even though an exact average of the possible futures keeps it (Sections 4, Table 1). Test 2 trains the same agent inside each model's imagination: agents trained in the flow model succeed while those trained in the JEPA largely fail, although the same agent trained on real experience through the JEPA's decoder does well (Section 5, Table 5).

Critical strengths

The paper's core strength is that it diagnoses a problem rather than proposing a method, and its experiments are built to test that diagnosis.

1. Problem: it names a real tension.

  • An objective that rewards predictability can throw away the part of the state an agent controls, which is often the least predictable part. The paper explains why: the encoder drops the state, or the rollout loses it, and the first has to be fixed before the second [Section 1; Section 2].

2. Approach: the tests separate the two mechanisms.

  • Sources of randomness are switched on one at a time, and the JEPA and a generative model are compared on the same data. The encoder is tested separately from the predictor: a flow predictor is put on the same frozen encoder, and an exact average is compared with the trained predictor [Section 3; Section 4].

3. Evidence: models are judged by whether an agent can learn to act inside them.

  • The paper scores each model by what an agent learns inside it, not by probe scores, and a control trains the same agent on real experience through the model's decoder. This ties each failure to decisions and separates "the state is missing" from "the imagination is poor" [Section 5].
Critical weaknesses

1. The known repair for mechanism 1, an action-prediction term, is never tested.

  • Mechanism 1 rests on "the controlled state, which nothing asks the encoder to keep" [p.1, Section 1]. An inverse-dynamics term asks exactly that, and the paper never tests one (no occurrence of "inverse", "IDM" or "action prediction" outside a proof).
  • Prior work the paper cites points the other way: Sobal et al. (2022) find an inverse-dynamics model "has great performance in all cases" under both per-frame and fixed noise, and in PLDM, a jointly trained JEPA, removing that term drops Diverse Maze success from 98.7 to 75.5 [cited p.2, p.9, Section 6].
  • Without this test, the title's "JEPAs" and the Outlook's call for "a principle for deciding what to keep" [p.9, Section 7] cover a class whose standard variant may not fail.
  • Fix: add an inverse-dynamics term to Equation 1 and rerun the per-frame wind world and the gates (three seeds each, minutes per fit [p.4, Section 3]).

2. The behavioral results rest on one fitted model per world.

  • "Each behavioral result rests on one trained model per world with three agent seeds, and the black gate's JEPA was chosen during development" [p.9, Section 7].
  • The wind failure behind mechanism 1 depends on regularizer weight and seed: at λ = 0.03 one of three fits keeps the cube [Appendix J, Table 9].
  • The swap behind the claimed order, representation before rollout, is scored with Test 1 only [p.9, Section 7].
  • These are disclosed, but the Abstract and finding 3 ("only the generative model's imagination teaches") still read as general results.
  • Fix: train three world models per world for Test 2, and run Test 2 on the swap.
Questions for authors

I don't have any specific questions. It would help if the authors could test a JEPA with an inverse-dynamics term (Weakness 1) and repeat Test 2 across several fitted world models (Weakness 2).

1 comment
AuthorsOpenReview ↗

Response to Reviewer 2FKk

You asked for two experiments. Both are in the revision, and the first one changed what the paper says.

Inverse dynamics. You were right that we could not make a claim about JEPAs without testing their standard repair. It works as the papers you cite say: under hidden drift the agent's position becomes readable from the latent (R² = 0.99) and mean survival rises from 0 to 71%. It does not help with the wall. In the four obstacle worlds, with three fitted models each, no agent survives more than 4% of episodes (Table 1). Proposition 1 gives the reason. The wall moves independently of the recorded actions, so encoding the wall cannot lower the error of predicting those actions. Inverse dynamics keeps what the agent controls, and the wall is what the agent does not control.

So the standard variant does repair part of the failure, as you suspected, and the paper now says so. The claim that remains is about external random events. We also tried a stronger recipe: a stopped target, inverse dynamics and task heads together. Its encoder keeps both the agent and the wall, and a policy trained on real transitions through that encoder survives 73 to 99% in the four obstacle worlds. Agents trained inside the model stay below 9%.

Several fitted models, and agents on the swap. Table 1 gives the mean and a 95% interval over independently fitted world models, each with its own agent. A few cells have fewer than three fits and are marked. The predictor swap is now scored by agents: with a flow-matching predictor on frozen stopped-target encoders they survive 98% on the fixed schedule and 72% with random arrivals, against 1% and 3% with the encoders' own predictors. The flow-matching predictor differs from the JEPA's predictor in more than its loss, so we also trained the flow predictor's network with a squared loss on the same latents. On three refitted encoders it gives 10% where flow matching gives 97% (Appendix C.7).

You also noted that the hidden-drift failure depended on the regularizer weight and the seed. It depends on the learning rate too: at a three times lower rate the plain JEPA keeps the agent and survives hidden drift in 79% of episodes. The obstacle failure does not move, and every fit at that rate remains below 5% (Section 3.2).

Comments on the paper

1 comment
AuthorsOpenReview ↗

Response to all reviewers

Thank you for the reviews. All three asked, in different ways, what the paper claims and whether its comparisons can carry that claim. The submitted version listed two mechanisms and a large number of measurements. The revision makes one claim and tests it one way.

The claim is that a JEPA trained end to end leaves out of its predictions the events an agent has to react to, and that the cause is its training objective, a regression loss against a target that the model itself produces. The test is to train an agent only inside the world model and then count how often it survives in the real environment.

We ran the experiments the reviews asked for. Inverse dynamics, reward and termination heads, a stopped target gradient, a moving-average teacher and LeWAM's action objective each went into LeWorldModel's recipe individually, and we also fitted one combined variant with a stopped target, inverse dynamics and task heads. Some of them repair the agent's own position under hidden drift. None of them gets an agent past a wall whose opening or arrival time is random: in 81 fitted JEPAs no agent survives more than 9% of episodes. A reconstruction tokenizer with a flow-matching predictor and task heads, trained on the same data, passes all five worlds with 72 to 94% mean survival (Table 1).

The new experiments also locate the failure. On the fixed schedule the LeWorldModel encoder still carries the wall, and a policy trained on real transitions through that encoder survives 95% of episodes. The model's own forecast has no wall. Two choices are responsible, the gradient through the prediction target and the regression predictor, and both have to be reversed: agents survive 3% inside LeWorldModel, 1% with only the target gradient stopped, 2% with only a flow-matching predictor, and 98% with both (Figure 4). With inverse dynamics and task heads added as well, a model built on a JEPA encoder passes all five worlds and matches the reconstruction model. It needs four changes to get there, and two of them are the reconstruction model's own predictor and heads. This is why we recommend reconstruction with a sampling predictor and task supervision as the default.

Results are now means over independently fitted world models with 95% intervals, and Appendix C.1 lists every fit.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.