LS-JEPA: Learning Long Short-Term Dynamics for Fast Latent Planning

Joint-Embedding Predictive Architecture (JEPA) offers a promising route to model-based control by predicting future latent dynamics. Current approaches mainly adopt one-step prediction objectives, which fail to capture the long-term latent transitions for action recovery. In this work, we introduce LS-JEPA, an end-to-end latent world model that jointly learns forward and inverse dynamics across Long Short-term temporal scales. Specifically, LS-JEPA combines one-step prediction with fixed-horizon autoregressive rollout, while an endpoint-conditioned action decoder reconstructs both individual actions and multi-step action sequences from latent displacements. These complementary objectives encourage predictive, action-relevant representations and directly connect latent transitions to control. Furthermore, the learned one-step and multi-step inverse mappings can be naturally used for fast latent planning during the inference time. We propose a Test-Time Replanning (TTR) strategy in which a goal-directed action sequence is decoded and then a one-step correction from a fresh observation is applied to refine the action. Extensive experiments are conducted to validate our proposed approach. The results show that LS-JEPA achieves the state-of-the-art results with 96.00%, 100.00%, 99.67%, and 100.00% on Push-T, OGBench-Cube, Reacher, and Two-Room, respectively, yielding a macro-average of 98.92% with only 11 ms of planning time per decision. These results suggest that jointly learning Long Short-Term forward and inverse dynamics with our proposed TTR strategy enjoys both the accuracy and efficiency for latent planning. Code is available at https://anonymous.4open.science/r/LS-JEPA-E034/.