- Rating
- 1
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 3
- Soundness
- 1
Read review
Paper summary
Summary of the paper
This paper is based on the energy-based model and shows mathematical identities connecting
- a score function of the statistical model in the conventional statistical sense ,
- and the corresponding loss of score-matching objective of the Stein's score
through the overdamped Langevin diffusion dynamics in Theorem 1. In addition, they show some identities related to stationarity of the burn-in time of the Markov chains defined by the above Langevin diffusion dynamics. They validate their proposals in the numerical experiments.
Contributions
The contributions of this paper are
- theoretical derivation of the identities connecting score functions and score-matching objective,
- stationarity analysis of the Langevin diffusion dynamics.
Validation
The authors validate the formulation by running some numerical experiments:
- checking gradient identity of = the Gaussian restricted Boltzmann machine (GRBM)
- with MNIST dataset (through numerics), and
- with synthetic 2d data (through optimization paths),
- checking burn-in time estimates of = the diagonal gaussian, the Gaussian restricted Boltzmann machine (GRBM)
- with data generated by another diagonal gaussian,
- training EBM by proposed algorithm called PISM.
Critical strengths
While interesting theorems such as Theorem 1, which directly link score matching to EBM, remain intriguing, modern diffusion models predominantly employ time-dependent score learning. In such cases, the contributions of this paper appear limited. (In Section 4.2, it seems only to include time-dependent scale factors as the only degree of freedom across the entire model.)
Critical weaknesses
- The manuscript contains multiple unclear points with numerical experiments.
- While this paper fundamentally aims to establish connections between score matching and energy-based models, modern diffusion models typically incorporate time dependence in their implementations. Sections 4.2 and 5.3 describe connections with these approaches, but it appears that only time-dependent scale factors can be included as the only degree of freedom across the entire model. Consequently, the extent to which this paper contributes to the performance analysis and improvement of modern generative models remains unclear.
Questions for authors
Question regarding the aforementioned weakness
- On numerical experiments:
- The left figure in Figure 3 appears to be a contour plot of the loss landscape, but which loss function? Additionally, if there is a symmetry , then the actual contour should possess this symmetry. Why does it appear that the symmetry is somewhat slightly broken in the figure?
- The section around failure modes mentions that the proposed method cannot resolve the issue. While integrity is acknowledged, what exactly is being claimed?
- On algorithm1:
- It looks exactly like training a regular diffusion model - where's the difference?
General questions/comments
- In the context of diffusion models, I've seen reports that restricting the model to conservative forces (i.e., total derivatives) results in performance degradation (c.f., arXiv:2209.12753, arXiv:2402.03845.) From this paper, isn't there any comment about this?
- In Theorem 1, this seems natural to me, but there's one point: while the left-hand side has , the right-hand side appears to lack that dependency. How should we interpret this? Does it mean ?
- By the way, the legends in Figures 4(b) and 4(c) overlap with the graphs, making them difficult to read.