- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This work studies how switching from the original Ornstein–Uhlenbeck process of diffusion models to higher-order Langevin dynamics can mitigate memorization without compromising the quality of the generated data. It builds on a recent line of work that studies memorization in diffusion models (Kadkhodaie et al. 2024, Bonnaire et al. 2025, ) and how to prevent it. One of the main methods has been early stopping of the training dynamics (Bonnaire et al. 2025); however, in practice it is hard to know when to stop. In parallel, another line of work (Sterling & Bugallo 2025, Sterling et al. 2025) studied the consequences of replacing the process that transports the data distribution to Gaussian white noise, usually an Ornstein–Uhlenbeck process, with higher-order Langevin dynamics (HOLD). For HOLD of degree , one needs to introduce auxiliary variables representing higher-order derivatives (velocity, acceleration, etc.). For the generative dynamics, one only needs to learn the score with respect to the highest-order derivative.
First, the authors derive closed forms for the trajectories that are solutions of HOLD and for the optimal empirical score. The authors claim that the HOLD empirical score has a wider Non-Uniform Fourier Transform (NUFT) than the first-order OU empirical score, and is thus more subject to spectral bias (Rahaman et al. 2019), and that this is why memorization is delayed.
They then empirically study the effect of HOLD on memorization. They train UNets on three datasets (CelebA, Animal Faces and CIFAR-10) and study the evolution of the FID and of the fraction of memorized data points as a function of the number of training epochs, for different orders of HOLD. They show that increasing delays the onset of memorization without reducing quality.
Contributions:
- Derive the law of the solutions of HOLD and a formula for the empirical score.
- Show that increasing the order of HOLD delays memorization without degrading the FID of the generated images.
Critical strengths
-
The authors identify a new way to mitigate memorization in diffusion models by changing the generative dynamics and the objective one learns.
-
I find that the Pareto curve of versus FID is an effective way to compare the performance of generative models, and this is the first time I have seen it.
Critical weaknesses
-
Superficiality: even if the results are sound and supported both empirically and analytically, there are not many of them, and there are several interesting questions and "low-hanging fruits" that would improve this paper (see the Questions For Authors section).
-
The authors claim that increasing the order of HOLD moves the Pareto curves to the left, but in several figures this is not the case between and (Fig. 2(b), Fig. 4(b), Fig. 6(b)). Maybe I misunderstood how to read the plots, so I look forward to the authors' answer.
Questions for authors
- This work focuses on the generative process of diffusion models, but what about flow matching, rectified flows, or one-step methods, which are closer to the state of the art than DDPM? Can the method proposed by the authors be applied to these frameworks?
- In Figs. 2(b), 4(b) and 6(b), the setting with the best Pareto curve is , which is better than . Do you have an explanation? The caption of Figure 2 is clearly wrong: "Using HOLD and increasing the model order helps move the respective Pareto frontiers to the left".
- In Section 3.2, the authors compare the NUFT of the HOLD and OU empirical scores. This is an important argument for why HOLD delays memorization, yet the only figure illustrating it is in the appendix. I think this figure should be in the main text.
- Why is the FID plotted on a log scale in Fig. 2? It ranges from 40 to 200, which is barely one decade.
- Since the FID curves are a bit noisy, the authors could average them over several training runs. Would it change the picture?
- Now that the generative dynamics has changed, how much more costly is the new one compared to DDIM or DDPM?
- Does the scaling with the dataset size found by Bonnaire et al. 2025 change with the order of the dynamics?
2 comments
I thank the authors for their detailed answer to my questions.
I have some additional questions:
-
The authors said that as increases, the number of epochs needed to memorize increases. Bonnaire et al. (2025) showed that, for vanilla OU diffusion models, this timescale scales linearly with the dataset size . The authors said that increasing delays this timescale. Does it only increase the constant, with the scaling still linear in , or is there a new super-linear timescale?
-
Is the conclusion of your work that is the best trade-off between quality, memorization and compute (number of training epochs + inference steps)? For , does training become slower and slower?
The updates to the manuscript have improved the submission, but my main problem is still present, namely that I do not believe there is enough substance in the paper.
I will thus for now maintain my score because I find that the contribution is just incremental and I do not believe there are enough theoretical insights to be gained, although it is still an interesting research question.
Thank you for your constructive feedback and questions. We address each point below:
Question 1: The HOLD methodology is fundamentally designed for diffusion models. We are especially interested in merging HOLD into the formulation of Flow Map Matching, which is capable of one-step generation, but it remains future work. One key benefit of HOLD in this context is that the forward process constitutes a ``linear time-invariant'' system. We are therefore able to analyze it with classical signals and systems theory. Classical flow matching and one step models do not, in general, impose this LTI structure.
Question 2: You are absolutely right! I meant to say both orders n=2,3 outperform the VPSDE baseline; thank you for catching this. We will correct the caption of Figure 2 accordingly. Note however that there are several instances when n=3 does not memorize at all. In this case, its Pareto frontier reduces to a single point, which does fall to the left of n=2 in a few places. The reason the n=3 curves do not typically fall to the left of n=2 curves can be understood from Figures 2(a), 4(a), and 6(a). The n=3 curve takes longer to converge because third-order dynamics are more intricate than second-order dynamics, so the corresponding Pareto curves compare roughly steady-state n=3 points against transient n=2 points much earlier in training.
Question 3: Thank you for pointing this out. We put this figure in the appendix due to space constraints, but will move this figure to the main text.
Question 4: We adopted a log scale for FID in early experiments because FID and Fmem were visually colliding on our old plots. We agree the log axis is unnecessary; we will switch to a linear scale in the revision.
Question 5: The FID curve noise could be reduced by either averaging over separate training runs, or by increasing the number of samples collected at every 1000 training steps. We generate 1024 samples at every analyzed step. FID is typically computed from 10,000 samples; doing so at every analyzed step (roughly 2000 time points over the 2 million training iterations) would have taken an estimated 10--20 days of additional compute per experiment. We did not have enough time or compute to do this. Furthermore, we do not expect increasing the sampling size to change the qualitative picture, since Fmem is already reported with 95% confidence intervals and the curves are clearly far apart from each other with similar FID convergences.
Question 6: The HOLD method only changes the number of inputs in the first convolution layer and does not increase network depth. Practically speaking, in our CelebA example, the VPSDE network possesses approximately 4.024 million parameters, where HOLD n=2,3 require approximately 4.025 and 4.026 million parameters respectively. Additionally, all methods use the same number of discretization steps during sampling. Hence HOLD is not meaningfully more expensive to sample than a standard first-order diffusion model at a matched step count.
Question 7: Yes; the scaling of memorization onset with dataset size does change with the order n of the dynamics. We did not discuss this in the main manuscript, but we will emphasize it in the revision: higher-order dynamics require more training data before memorization sets in, and the gap between HOLD and the VPSDE (and between n=3 and n=2) widens as the training set grows.