Mitigating Diffusion Model Memorization with Higher Order Langevin Dynamics

Diffusion/score-based models have emerged as powerful generative models capable of generating high-quality samples that mimic the distribution of training data. However, these models have been observed to reproduce training samples, a phenomenon known as "memorization," potentially violating copyright and privacy. In this paper, we study the effect of Higher-Order Langevin Dynamics (HOLD) on this phenomenon. HOLD diffusion processes introduce auxiliary variables; if the data variable is interpreted as "position," then the auxiliary variables can be interpreted as "velocity" and "acceleration," depending on the chosen order of the model. They were originally proposed based on the intuition that these auxiliary variables regularize the trajectories of the data variable by implicitly imposing additional dynamical constraints. Our work provides, to our knowledge, the first theoretical characterization of this time-domain regularization effect of HOLD. Specifically, we show that in HOLD, the dynamics of the data variable are governed by a low-pass-filtered version of the learned score function, with smoothness increasing with the order of HOLD. We then analyze the optimal empirical score of HOLD and using spectral analysis reveal that, at low noise levels, it is hard to learn. Together, our results explain how increasing the model order mitigates memorization. Finally, we present an empirical study on real-world data that supports our theory and highlights this distinct advantage of HOLD over standard diffusion models in practice.