Guiding Diffusion Samplers with Black-Box Optimization Solvers

From robot motion planning to scientific inverse problems, generative tasks often require optimizing target objectives while adhering to a complex data-driven prior. Diffusion models excel as such priors, but guiding them to optimize targets or obey constraints remains challenging, even with powerful optimization solvers available. When integrated into diffusion sampling, these solvers often distort the sampling trajectory with their hard projections. We propose Proximal Diffusion Guidance (ProxDG), a training-free method that cleanly integrates solvers as soft guidance during diffusion sampling. Specifically, ProxDG guides samples using the gradient of a time-varying Moreau envelope. This envelope smooths the original objective without shifting its optima, providing well-behaved guidance even under hard constraints. Importantly, computing this guidance reduces to a proximal subproblem that admits powerful, problem-specific solvers. With this proximal scheme, ProxDG samples along a continuous probability path to the target posterior, supporting both stochastic and deterministic sampling with exact constraint enforcement at the final step. We demonstrate in both synthetic and real-world experiments that ProxDG effectively leverages off-the-shelf solvers to significantly outperform training-free baselines.