- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
The paper asks which forward noise schedule minimises the KL error of the discretised reverse sampler. The authors combine Girsanov's theorem and a scalar ODE for the Fisher information. Together these give a KL upper bound, which reduces the problem to a scalar function of time. Proofs are provided in the appendix. The experiments compare an 8-parameter ACS instance against VP-linear, VP-sigmoid, VP-cosine and EDM schedules. Each schedule is tuned once with 32 Optuna trials on CIFAR-10 and ImageNet-64 at NFE 10 and 50. The tuned schedules are then transferred by SNR matching to pretrained EDM/EDM2 models on held-out datasets. In this setting ACS is reported to be competitive in FID-50k across the efficient sampling range of 10 to 50 NFE.
The paper provides three theorems
(i) An error bound for the sampler that depends explicitly on the chosen noise schedule. The bound is expressed through a single quantity that tracks how much structure remains in the noisy data over time (Thm. 1).
(ii) The schedule that minimises this bound (Thm. 2). Under this schedule, the remaining structure is removed at a steady, constant pace. It matches the best error rate currently known: the error grows with the data dimension and shrinks with the number of sampling steps. However, it cannot be used directly in practice, because constructing it requires knowledge of the data distribution that is not available.
(iii) For simple data distributions (well-behaved single-peaked distributions and mixtures of round, equally wide Gaussian clusters), a concrete family of schedules, called ACS, that can be written down explicitly. ACS achieves the same rate, provided problem-specific constants are ignored (Thm. 3). The family includes the shapes of several schedules already used in practice (VE-exponential, VP-sigmoid and VP-linear). It also prescribes how their settings should change with the number of sampling steps (Cor. 1).
Critical strengths
(i) The paper tracks the Fisher information, which is a single scalar that follows a simple ODE instead of optimising over evolving densities. The schedule design then becomes a one dimensional optimal control problem in theory with a clean closed-form solution (Thms. 1-2).
(ii) Prior sampling error guarantees rely on constructions rarely used in practice. Corollary 1 shows that VE-exponential, VP-sigmoid and VP-linear schedules reach the same rate if their parameters scale with the number of sampling steps. For example, the range of VP-linear should grow logarithmically with the step budget rather than stay fixed.
Critical weaknesses
(i) The paper does discuss related work, but I could not find a discussion of two closely related works in particular. [1] derive a KL bound that depends explicitly on a time-dependent noise schedule. [2] optimise the sampling time grid by minimising a Girsanov-based KL upper bound on the discretisation error. The paper itself claims that schedule design and grid design are equivalent (App. B.3), so these works address the same problem.
(ii) The headline guarantee may be weaker than stated, and the experiments do not test it. Theorem 3 reaches sampling error guarantees only “ignoring problem constants”, and its constant can grow with dimension. For Gaussian mixtures whose cluster means are far apart in high dimension, the constant grows linearly with d (Lemma 7). The paper's own bound then should become roughly , which to my understanding is no better than the older bounds it aims to improve on. Furthermore, while showing clear trends, the experiments show no error bars, and can not fully isolate the components derived by theory. It is unclear if the empirical experiments are statistically significant.
[1] Strasman, S., Ocello, A., Boyer, C., Corff, S.L., Lemaire, V. (2024). An analysis of the noise schedule for score-based generative models. arXiv preprint arXiv:2402.04650.
[2] Sabour, A., Fidler, S., Kreis, K. (2024). Align Your Steps: Optimizing Sampling Schedules in Diffusion Models. arXiv preprint arXiv:2404.14507.
Questions for authors
(1) How does your work differ from [1,2]? Please add them to your related work discussion, if possible.
(2) How large is the constant in Theorem 3 for realistic data? For a Gaussian mixture whose clusters are far apart in high dimension, does the bound still scale like , or closer to ?
(3) Regarding Theorem 3, "where satisfies ". This constraint on in Theorem 3 involves the Fisher information of the data distribution. How does this fit the claim that parameters can be chosen without knowing the Fisher information? How large must be in practice for the guarantee to apply?
(4) Which part of ACS is derived from the theory, and which is tuned freely? With 8 tuned parameters against 1-3 for the baselines, it is unclear whether the gains come from the theory derived step-dependence or from extra flexibility.
[1] Strasman, S., Ocello, A., Boyer, C., Corff, S.L., Lemaire, V. (2024). An analysis of the noise schedule for score-based generative models. arXiv preprint arXiv:2402.04650.
[2] Sabour, A., Fidler, S., Kreis, K. (2024). Align Your Steps: Optimizing Sampling Schedules in Diffusion Models. arXiv preprint arXiv:2404.14507.