Breaking the Rotation Barrier in JEPA World Models with Anisotropic Predictors

Joint-Embedding Predictive Architectures (JEPAs) learn world models by predicting in representation space, but their predictor can absorb any rotation of that space without changing the loss, leaving physical factors mixed across coordinates and hard to monitor or control individually. Recovering one factor per coordinate requires the right subspace, which contains the factors, and the right basis, which gives each factor its own coordinate. We break this rotation barrier with a minimal change, replacing the dense predictor by an anisotropic predictor (**AP-JEPA**), and prove that its loss splits exactly into two terms: a rotation-invariant ***dense-predictor term***, which selects the subspace, and a ***cross-talk term***, which selects the basis and aligns coordinates with factors that evolve on distinct timescales. However, the encoder can produce features that are more predictable than genuine factors, such as the square of a slow factor, so the selected subspace may miss some factors. We characterize when two existing regularizers, whitening and SIGReg, rule out such features, and propose a new regularizer, PairSIGReg, which constrains consecutive representations jointly and recovers the factor subspace for every spectrum of distinct timescales. Across six dynamical and image benchmarks, AP-JEPA with PairSIGReg raises the average mean correlation coefficient (MCC) from $63.6\%$ for the strongest baseline to $77.8\%$, while also achieving the highest average linear readout ($90.2\%$). Beyond training from scratch, AP-JEPA also applies to existing JEPAs trained with dense predictors: since the cross-talk term depends only on the basis, minimizing it alone yields a closed-form change of coordinates that raises the single-coordinate $R^2$ of released I-JEPA and V-JEPA models from about $10\%$ to $81\%$ without information loss.