DPMPC: Learning Diffusion Priors for Model Predictive Control

Model predictive control (MPC) has achieved notable success in continuous control. MPC often uses a policy prior to propose candidate trajectories and to learn the value function. When the prior differs from the executed planner, this actor divergence biases the value estimates of candidate trajectories. The commonly used Gaussian prior cannot represent the non-Gaussian action distributions induced by stochastic planning, leaving a residual divergence. To address this, we propose DPMPC (Diffusion Priors for Model Predictive Control), which trains an expressive diffusion prior to match the planner's action distribution. Generalized lazy reanalyze estimates this distribution from multiple recent MPC realizations stored per replay state, without rerunning MPC at every policy update. An advantage-weighted score-matching objective then fits the prior to this estimate while emphasizing high-value actions. DPMPC achieves state-of-the-art asymptotic performance and sample efficiency across 14 continuous-control tasks from DMControl, MyoSuite, and HumanoidBench. Further analysis shows that DPMPC yields more accurate value estimates than Gaussian-prior baselines.