Learning Physical Dynamics through Relational Prediction

Predicting physical trajectories in unfamiliar configurations requires reusing local interactions. Trajectory supervision evaluates their combined effect on motion, so fitting training trajectories need not capture how individual relations evolve. We introduce **HORME**, which trains the same pair features for motion computation and prediction of relational evolution. To supervise this evolution during recursive prediction, we propose Reference-Anchored Joint-Embedding Predictive Architecture (RA-JEPA). It matches predicted and observed feature changes pair by pair in a shared space, retaining initial features as a fixed reference. The forecasts condition physical pair features, allowing trajectory errors to train the predictor. Computing displacement further requires resolving when acceleration acts within each step. To capture timing effects missed by sparse acceleration samples, we propose Temporal Response Decomposition (TRD), separating displacement contributions from mean acceleration and its timing. A readout shared across integration stages corrects the timing-dependent component while preserving the current step's velocity update. HORME achieves the lowest trajectory root mean square error (RMSE) among compared methods on I-PHYRE and gravitational N-body, including unseen games, larger systems, and extended horizons.