Learning Latent Waypoints for World-Model Planning

Latent world models enable efficient planning by evaluating candidate actions in compact representation spaces, but terminal goal-matching costs provide little information about useful intermediate states. We propose Latent Residual Waypoints, a lightweight method for generating intermediate waypoints between start and goal latents. Given a frozen, pretrained world-model encoder, our method predicts encoded intermediate frames by learning a residual correction to latent linear interpolation. During planning, we generate a sequence of latent waypoints recursively. These are incorporated through a minimum-over-time waypoint cost that encourages trajectories to pass near future waypoints while still optimizing for the final goal. This enables effective planning in difficult settings, such as planning with budget constraints. A planner solving episodes of the Two-Rooms and Cube tasks under a constrained CEM budget can outperform full-budget final-goal planning. At matched budgets, the performance deltas are even more pronounced. We also find that our method improves Le-WorldModel's CEM planning performance across navigation and manipulation tasks and outperforms simpler waypoint methods like linear interpolation and nearest-neighbor snapping. We extend this to other world models and show improvements on gradient-based closed-loop MPC planning. Though our waypoints generally improve mean success rate compared to final-goal planning, they yield fewer gains on some easier tasks. Qualitative analysis of visualized latent waypoints and nearest-neighbor analyses further show that Latent Residual Waypoints remain closer to realistic trajectory latents than those produced by naive interpolation.