Latent Space Planning with Sequential Monte Carlo

Planning with learned world models often relies on trajectory optimization on latent rollouts of action sequences through differentiable dynamics. Stochastic variants inject noise into these updates to escape spurious basins, but unlike zero-order planners, they do not maintain a population. In contrast, sequential Monte Carlo (SMC) performs population-level inference, where importance weighting and resampling progressively steer a candidate population toward regions favored by the target distribution and can be combined with gradient frameworks. In this work, we propose latent-space planning as SMC, enabling inference through steering and optimization. Because the cost determines where the population is steered, we also study how costs influence planning, specifically, the ($L_2$) objective typically used by planners and a structured cost inspired by local progress measures from classical planners. Across five environments and two world models, we find that SMC improves goal-reaching success over zero-order, gradient-based, and stochastic gradient-based planners, with the effect of the cost structure becoming more pronounced in more complex settings. We find that SMC improves average goal-reaching success by upto $19$, $23$ and $33$ percentage points over zero-order, gradient-based and stochastic gradient-based planners, respectively.