Latent Metrics for Planning with World Models

Joint-Embedding Predictive Architectures (JEPAs) enable visual planning by predicting the future outcomes of candidate actions directly in latent space. However, existing JEPA planners typically evaluate candidate actions using Euclidean distance between the predicted outcome and the goal. This treats all latent directions equally, despite differences in their sensitivity to actions and in the model's prediction accuracy along them. We introduce Latent Metrics for Planning with World Models (LENS). LENS identifies the action-response fidelity of different latent directions from the action-conditioned dynamics already learned by a pretrained world model, while keeping both the encoder and predictor frozen. Specifically, LENS derives action-response fidelity from the consistency of action-conditioned variation between real and predicted futures together with the accuracy of the corresponding future predictions, and uses this fidelity to determine the relative influence of different latent directions in planning. Directions with higher action-response fidelity receive greater relative weight in the planning metric. The resulting metric is used both to evaluate predicted futures and to guide sampling-based search, without retraining the world model. Across four benchmarks, LENS improves average success rate from 85.8% to 95.8%. With candidate actions and world-model predictions fixed, it increases the average Spearman correlation between model rankings and real outcomes from 0.540 to 0.696. These results show that the action-conditioned dynamics already learned by a world model can provide not only a basis for predicting the future, but also a basis for determining the relative role of different latent directions in planning.