acceptodds
Under review as a conference paper at ICLR 2027

Global Convergence of Gradient Descent for Score Matching in Gaussian Mixtures via Reverse Fisher Divergence

Abstract

The score matching problem is a central training objective in modern generative modeling, diffusion models, fitting unnormalized statistical models, and inverse problems. A standard approach is to minimize the forward Fisher divergence, where the expectation is taken with respect to the teacher distribution. However, recent results show that even in simple Gaussian mixture model settings, this objective can lead to undesirable and initialization-dependent convergence behavior. In this paper, we study an alternative objective: the reverse Fisher divergence, where the expectation is taken with respect to the student distribution. We analyze gradient descent (GD) for fitting Gaussian mixture models and show that this change in the objective leads to significantly better optimization properties. First, when the teacher distribution is a single Gaussian and the student is a Gaussian mixture model with fixed weights and identity covariances, we prove the global convergence of GD from arbitrary initializations. Second, we extend the analysis to the case where the teacher is also a Gaussian mixture model and prove global convergence guarantees under a global random initialization scheme and a -separation assumption on the target means. In particular, with high probability, each student component converges near its closest teacher component, and we provide conditions under which the student distribution converges in total variation distance. Our proofs rely on a new Lyapunov-based analysis of the gradient descent dynamics, showing that the reverse Fisher divergence has a much more favorable optimization landscape than the forward Fisher divergence.

Primary area:
optimization
Keywords:
score matching, global convergence, Gaussian Mixtures, Fisher Divergence

Reviews

Among:
Shape:
1234this paper, Top 1%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.67 over 3 reviews, above 99% of the others.
Reviewer_rnRNOpenReview ↗
Rating
4
Confidence
3
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

This paper studies the convergence of score matching using reverse Fisher divergence loss. It shows that, unlike the common forward divergence, this reverse variant allows global convergence guarantees for Gaussian mixtures. It concerns fitting the means of a teacher GMM, and the results contain the cases of fitting a single Gaussian and multiple Gaussian components, as well as an analysis on the TV distance between the student and teacher distributions.

Critical strengths
  • Using reverse Fisher divergence for score matching is an interesting idea, and the resulting global convergence guarantees are surprising and significant.
  • The presentation is very clear, with good intuitions about the theoretical results and proof techniques.
  • Figure 2 shows the difference between forward and reverse Fisher divergence and corroborates the presented theory well.
Critical weaknesses
  • My major concern is in the practical relevance of reverse Fisher divergence. Both forward and reverse Fisher divergences are defined using the score of the unknown teacher distribution. The key observation that enables score matching is that the forward divergence can be rewritten in a form that does not depend on the teacher's score, e.g., in Hyvärinen's original score matching [1] and modern denoising score matching [2]. Does the reverse Fisher divergence admit a similar reformulation? Can the reverse Fisher divergence serve as a practical alternative to forward divergence in statistical estimation or training diffusion models?

[1] A. Hyvärinen, "Estimation of Non-Normalized Statistical Models by Score Matching," Journal of Machine Learning Research, vol. 6, no. 24, pp. 695–709, 2005. [2] P. Vincent, "A Connection Between Score Matching and Denoising Autoencoders," Neural Computation, vol. 23, no. 7, pp. 1661–1674, Jul. 2011.

Questions for authors
  • See weakness 1.
  • Can the authors comment on the impact of ρ\rho in Theorem 3.2? This value controls the failure probability in the high-probability bound. The theorem only explicitly characterizes the effect of ρ\rho on the initialization radius. How does it affect other assumptions and the iteration complexity?
Reviewer_YEurOpenReview ↗
Rating
4
Confidence
2
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

The paper studies gradient descent for fitting Gaussian mixtures using reverse Fisher divergence, which averages the squared score error over the model distribution rather than the data distribution. It proves convergence from arbitrary initialization when the target is a single Gaussian, and approximate component recovery for separated Gaussian-mixture targets under a specific random initialization. Numerical experiments illustrate improved optimization behavior compared with forward score matching. The guarantees assume exact population gradients, fixed equal mixture weights, and identity covariances; recovering the full target distribution requires additional conditions.

Critical strengths
  • the paper give a strong single-target guarantee: for a single Gaussian target, all student means converge from arbitrary initialization, contrasting with known failures of forward score matching;
  • the proof of this result is interesting: relying on a Lyapunov analysis, the authors give quantitative convergence rates through a positive-semidefinite matrix representation and a coercivity estimate;
  • possible extension to multiple targets: the paper goes beyond the single-Gaussian case, establishing approximate component recovery under the assumption of clear separation of the modes and random-initialization conditions.
Critical weaknesses
  • Restrictive model class. The analysis learns only mixture means, with fixed equal weights and identity covariances. Its relevance to more general mixtures and neural score models remains unclear.

  • Conditional “global” convergence for multiple targets. Unlike the single-target theorem, this result requires initialization on a sufficiently large sphere and strong separation between target means. The separation also depends on the requested accuracy.

  • Limited experimental scope. The low-dimensional examples illustrate the predicted behavior but do not establish robustness to noisy gradients, more general covariances, or unbalanced mixtures.

Questions for authors

While the theoretical results are interesting, I found it difficult to understand how they could inform practical training when only data samples are available. The following questions concern possible directions for bridging this gap.

Practical estimation: When the teacher score is unknown and only data samples are available, how could the reverse Fisher objective and its gradient be estimated? Is there a tractable reformulation analogous to classical score matching that avoids explicit access to the teacher score and its derivatives?

Stochastic gradients: Could the convergence analysis accommodate Monte Carlo gradients and errors in an estimated teacher score? Which parts of the proof would be most sensitive to these approximations?

Initialization: Could comparable guarantees hold under a more conventional initialization, without placing student means on a large sphere?

Reviewer_brCGOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper studies gradient descent on reverse Fisher divergence for Gaussian mixtures with fixed equal weights and identity covariances. For a single Gaussian teacher, it shows that all student means converge to the teacher mean from any initialization. For a mixture teacher, it gives guarantees under random initialization and separation assumptions, while accounting for possible errors in the component weights.

Critical strengths

Theorem 2.1 is the result I find most interesting. It shows convergence of every student mean from any starting point, addressing failure cases that occur with forward Fisher minimization. The proof also explains why convergence happens: a Lyapunov function tracks all the means and decreases at a controlled rate.

Critical weaknesses

The mixture result needs much stronger assumptions, including a large initialization radius and separation that depends on the desired accuracy. Even when the component locations are close to the teacher means, the distributional bound still contains 12∑a∣pa−1/m∣\frac{1}{2}\sum_a |p_a-1/m|, where pap_a is the initial-assignment probability for teacher component aa, among mm components. More student components alone do not guarantee that this term is small. The authors point this out in Section 3.2, so the mixture result has a narrower scope than the single-Gaussian result.

Questions for authors

What bounds on the unknown teacher parameters do we need to know before choosing the initialization radius?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.