Learning to Ignore What Matters: JEPAs in Stochastic Worlds

Joint-embedding predictive architectures (JEPAs) learn world models by predicting future representations rather than reconstructing observations. We show that this objective does not protect the information needed for control. We identify two failure mechanisms for jointly trained JEPAs in stochastic environments: the representation can discard the state that decisions depend on, and the predictor can lose decision-relevant features even when the representation keeps them. We test both by training agents entirely inside a JEPA world model and inside a generative world model with the same data and compute, then evaluating them in the real environment. The JEPA supports far weaker control: agents trained inside it survive up to 50 times less often and fail to learn multi-step behavior in Craftax. These failures are not caused by collapse, and rollouts that look correct can still teach agents to exploit errors in predicted rewards and terminations. Low latent prediction error and non-collapsed representations coexist with poor closed-loop performance, so neither is sufficient for control.