LFNT : Forecasting training trajectories in the latent space of weights

Forecasting neural network weights can accelerate training by skipping portions of the optimization trajectory, but existing approaches operate directly in weight space. Motivated by the success of learned latent representations in other domains, we propose forecasting optimization trajectories in a learned latent space of weights. We introduce **L**atent **F**orecasting of **N**eural **T**rajectories (**LFNT**, pronounced “elephant”), a three-stage approach that (i) learns a latent representation of model weights, (ii) aligns these representations using forecasting supervision, and (iii) trains a forecaster on the resulting latent trajectories. Across vision and language tasks, latent-space forecasting provides greater training speedups than weight space forecasting baselines, with speedups of up to 59%. Overall, the results show that the representation of optimization trajectories substantially affects forecasting performance, and that a learned latent space can provide a better alternative to forecasting directly in weight space.