- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper studies the contraction of Fisher divergence for both overdamped and underdamped diffusions. Under the assumption of exact score functions and regularity conditions on the target distribution, such as score Lipschitzness and sub-Gaussian tails, the authors establish exponential contraction of Fisher divergence. Experiments on Gaussian mixtures illustrate this contraction.
Critical strengths
- The paper provides a useful perspective on understanding the convergence of diffusion-based samplers through Fisher divergence. Its comparison between probability-flow ODEs and reverse-time SDEs helps clarify their different behaviors under initialization mismatch.
Critical weaknesses
- The practical implications of Fisher divergence contraction for diffusion-based samplers remain unclear. Even in the Gaussian-mixture experiments, the contraction of Fisher divergence does not necessarily imply accurate recovery of the target distribution. Moreover, the theoretical and experimental results mainly focus on initialization error, which is typically small in practical diffusion models.
- The theoretical setting is restrictive and lacks sufficient justification. The target distribution is assumed to have a Lipschitz score and sub-Gaussian tails, which seem stronger than the assumptions required in some existing KL/TV analyses[1,2] of diffusion models. The paper does not sufficiently discuss the practical relevance of these assumptions. Moreover, assuming exact scores or score errors controlled in also appears difficult to justify in practice.
[1] Chen H, Lee H, Lu J. Improved analysis of score-based generative modeling: User-friendly bounds under minimal smoothness assumptions [C]//International Conference on Machine Learning. PMLR, 2023: 4735-4763.
[2] Benton J, De Bortoli V, Doucet A, et al. Nearly $ d $-linear convergence bounds for diffusion models via stochastic localization[C]//International Conference on Learning Representations. 2024, 2024: 36916-36936.
Questions for authors
- Could the authors provide more discussion on the practical implications of Fisher divergence contraction? In particular, what advantages does Fisher divergence analysis offer over existing KL/TV analyses?
- Why do the experiments choose far from ? In practical diffusion models, the initial distribution for sampling is usually chosen to be close to the terminal distribution of the forward process.
4 comments
(Part 2 of 2, continued from our previous comment.)
Weakness 2(a): Strength of score regularity and tail assumptions
We have refined our proof techniques to remove the sub-Gaussian tail assumption, but the Lipschitz score assumption is still needed, and we agree that the discretized theorems have stronger data-side assumptions than several existing KL/TV analyses.
The reason is partly the quantity being controlled. Theorem 4.5 of the original submission (now Theorem 4.3 in the revision) bounds , an expectation under the numerical law. In contrast, many KL analyses of diffusion models use a direction of relative entropy and a path-space argument that place error expectations under the exact process and can exploit the OU structure under weaker score regularity. These are different targets and proof methods.
We will state that the current assumptions are sufficient rather than sharp, explain their exact role in Remark 4.5, and identify removing or weakening them as an open problem.
Weakness 2(b): Exact scores and learned-score error under the sampler law
This is a genuine limitation, and we agree it deserves greater prominence. The paper proves its principal results for exact scores. Remark 4.7 explains how an additional mean-square score error term can be incorporated when the error is controlled under the sampler law. However, the usual denoising-score-matching guarantee is under the exact noised-data marginal, and transferring that guarantee to the numerical law requires extra assumptions. An bound alone cannot exclude large score errors in regions that the numerical sampler overweights.
We will clarify this distinction and describe possible sufficient conditions for the transfer (for example, bounded error, or suitable exponential-moment bounds). The pretrained-DDPM experiment is useful empirical evidence that initialization-dependent bias can persist with learned scores. A complete theorem simultaneously treating realistic score estimation, discretization, and Fisher stationarity remains future work.
Thank you for the detailed clarifications.
While I understand that Fisher contraction does not necessarily imply distributional recovery, the practical significance of the theoretical results remains unclear to me. As shown in the Gaussian-mixture experiment, Fisher divergence becomes very small while KL and TV remain substantial. In practice, diffusion models are primarily evaluated by the quality and fidelity of generated samples, which Fisher contraction alone does not guarantee.
Could the authors further clarify what meaningful insights or guarantees these theoretical results provide for practical diffusion sampling, particularly with learned scores and discretization errors?
Thank you for the thoughtful follow-up. We agree that the practical significance of Fisher contraction deserves clarification. We provide the following additional argument connecting Fisher divergence to within-mode distributional accuracy. These findings clarify what Fisher contraction guarantees even when global KL and TV remain substantial.
1. Quantitative within-mode accuracy from Fisher divergence
Our results distinguish within-mode fidelity from global mode coverage. Although small Fisher divergence does not imply global distributional recovery, it can control within-mode distributional discrepancies under suitable local regularity conditions.
Partition the sample space into regions , representing modes or groups of similar samples. Let , , and , . The KL chain rule yields
Suppose each conditional target satisfies a log-Sobolev inequality on with constant , namely . Because is a constant multiple of within each region, the relative scores agree there and . Thus, writing ,
Pinsker's inequality and Cauchy–Schwarz also give
Hence Fisher divergence controls within-mode KL and TV when local log-Sobolev constants are moderate, even if the global distribution remains far from the target. Such local constants remain favorable for regions with a single mode (for an approximately Gaussian mode, the constant is of the order of the mode variance).
2. Concrete Gaussian-mixture evidence
In the Gaussian-mixture experiment, we can partition the real line at the midpoints between adjacent mode centers and divide it into 3 regions for 3 modes. Although terminal global KL and TV remain substantial (0.401 and 0.405), the weighted sum of conditional within-mode KL divergences is only approximately 0.005. Specifically,
Thus, approximately 99% of the residual global KL arises from incorrect mode weights, not from within-mode discrepancies. The small Fisher divergence therefore corresponds to substantial within-mode distributional accuracy even though global recovery is incomplete. Analogously, an unconditional image generator could produce convincing cats, dogs, and fish while sampling these categories in incorrect proportions.
3. Practical interpretation with learned scores and discretization
Our CelebA-HQ experiment (Appendix F.4) uses a pretrained DDPM with learned scores and discretization. An initialization shift of approximately 0.009 noise standard deviations per pixel yields visually realistic faces but noticeably biases the brightness distribution. This illustrates that plausible individual samples can conceal substantial global distributional bias.
It is empirically known that sample quality is decided mostly in the late, low-noise stages of generation, which is also where most samplers concentrate their steps. If a model is poorly trained or poorly scheduled at early stages, or the forward process is truncated too early, the resulting error acts as an initialization error for the rest of the process. Our theoretical results show that stochastic correction makes the generated samples locally indistinguishable from real data, even when this error is very large, and in doing so it can hide a severe bias in the sample proportions. This perspective complements global KL/TV convergence theory. It motivates evaluating both within-mode fidelity and global mode coverage when assessing diffusion samplers.
In summary, Fisher contraction can yield meaningful within-mode KL/TV accuracy, even when large errors are incurred at early generation stages. Together with our discretization and learned-score analysis, this offers a quantitative explanation of how local fidelity can coexist with global distributional bias.
We thank the reviewer for the careful assessment and constructive suggestions. We address each point below (Part 1 of 2).
Weakness 1(a) and Question 1: Practical meaning of Fisher contraction and comparison with KL/TV
We agree that Fisher contraction does not imply accurate recovery of the target distribution. This limitation is a central message of the paper, not a claim we seek to avoid. The relative Fisher divergence measures the L2(p) discrepancy between the scores of p and q; it is a first-order, local certificate. It can be small even when probability mass is incorrectly allocated among well-separated modes. Accordingly, Fisher analysis complements rather than replaces KL/TV guarantees.
Its distinct contribution is to describe how initialization-induced score mismatch, or errors induced by discretization or score estimation as explained below, evolves under the reverse dynamics. The exact reverse SDE satisfies an entropy dissipation identity without a functional inequality on the data and, under strong convexity of the chosen forward potential, an exponential Fisher contraction estimate. General KL/TV data-processing inequalities alone do not provide this Fisher-level contraction rate.
In the Gaussian-mixture experiment, the reported terminal KL and TV remain around 0.40 and 0.41, respectively, while the terminal relative Fisher divergence is only about 4.5×10−3, down from an initial value of 100. This demonstrates the gap between local stationarity and global fidelity. The CelebA-HQ experiment provides a high-dimensional illustration: visually plausible images can coexist with severe brightness-selection bias. Thus, the practical benefit of the Fisher viewpoint is a more precise characterization of what the stochastic dynamics can repair, and what they cannot certify.
Weakness 1(b) and Question 2: Why study nonstandard initializations?
We agree that standard generation typically starts from a tractable Gaussian chosen to approximate the terminal forward marginal, and that making this approximation accurate is important. The deliberately perturbed initializations in our experiments are stress tests designed to reveal the stability mechanism; they are not presented as representative of the default initialization used in every deployed model.
In the CelebA-HQ experiment, the mean shift is μ=±4⋅1/d. Each coordinate is shifted by only about 0.009 standard deviations for d≈2×105. Its purpose is to expose the sensitivity of the sampling bias. The same phenomenon is known in deployed models (Lin et al., 2024) even with standard normal initialization. Our theory explains why the reverse SDE does not repair it.
Initialization mismatch also provides a controlled model for perturbation propagation. If the exact reverse dynamics is resumed from a perturbed law at an intermediate time, the dissipation and contraction identities apply to the remaining exact flow. This observation motivates studying perturbations caused by numerical or score errors, even though it does not automatically establish a quantitative bound for their continuously accumulated effects. Guided image-editing methods such as SDEdit (Meng et al., 2022) also intentionally initialize from laws different from the unconditional terminal marginal, making this stability question relevant beyond standard unconditional sampling.
References
Lin, S., Liu, B., Li, J., and Yang, X. Common diffusion noise schedules and sample steps are flawed. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision (WACV), pp. 5404-5411, 2024. https://arxiv.org/abs/2305.08891
Meng, C., He, Y., Song, Y., Song, J., Wu, J., Zhu, J.-Y., and Ermon, S. SDEdit: Guided image synthesis and editing with stochastic differential equations. In International Conference on Learning Representations (ICLR), 2022. https://arxiv.org/abs/2108.01073