- Rating
- 4
- Confidence
- 3
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
This paper studies the convergence of score matching using reverse Fisher divergence loss. It shows that, unlike the common forward divergence, this reverse variant allows global convergence guarantees for Gaussian mixtures. It concerns fitting the means of a teacher GMM, and the results contain the cases of fitting a single Gaussian and multiple Gaussian components, as well as an analysis on the TV distance between the student and teacher distributions.
Critical strengths
- Using reverse Fisher divergence for score matching is an interesting idea, and the resulting global convergence guarantees are surprising and significant.
- The presentation is very clear, with good intuitions about the theoretical results and proof techniques.
- Figure 2 shows the difference between forward and reverse Fisher divergence and corroborates the presented theory well.
Critical weaknesses
- My major concern is in the practical relevance of reverse Fisher divergence. Both forward and reverse Fisher divergences are defined using the score of the unknown teacher distribution. The key observation that enables score matching is that the forward divergence can be rewritten in a form that does not depend on the teacher's score, e.g., in Hyvärinen's original score matching [1] and modern denoising score matching [2]. Does the reverse Fisher divergence admit a similar reformulation? Can the reverse Fisher divergence serve as a practical alternative to forward divergence in statistical estimation or training diffusion models?
[1] A. Hyvärinen, "Estimation of Non-Normalized Statistical Models by Score Matching," Journal of Machine Learning Research, vol. 6, no. 24, pp. 695–709, 2005. [2] P. Vincent, "A Connection Between Score Matching and Denoising Autoencoders," Neural Computation, vol. 23, no. 7, pp. 1661–1674, Jul. 2011.
Questions for authors
- See weakness 1.
- Can the authors comment on the impact of in Theorem 3.2? This value controls the failure probability in the high-probability bound. The theorem only explicitly characterizes the effect of on the initialization radius. How does it affect other assumptions and the iteration complexity?