- Rating
- 2
- Confidence
- 2
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
This paper relates entropy changes in diffusion processes to the scores learned by generative models. It derives an exact decomposition of a likelihood surrogate into entropy terms and an integrated squared score error, yielding a lower bound on the surrogate and an upper bound on data entropy. A separate residual describes the difference between the surrogate and probability-flow likelihood. Experiments cover synthetic distributions, finite Ising systems, and pretrained image models. The paper also introduces the entropy reduction ratio as a diagnostic and explores its use for choosing model size.
Critical strengths
-
The decomposition gives a clear interpretation of the bound’s slack as integrated squared score error. Deriving the data-entropy bound from the same identity provides a useful connection between score approximation and entropy estimation.
-
The synthetic and finite Ising experiments allow meaningful comparisons with independent entropy estimates or analytic targets. Measuring the difference between the surrogate and actual likelihood is also valuable, since it makes the limitations of transferring the bound to actual likelihood visible.
Critical weaknesses
-
The main theorem bounds a surrogate, not the model’s actual negative log-likelihood. Because the residual connecting the two has no fixed sign, the theorem does not establish the general likelihood limit suggested in the introduction. Appendix C.4 provides sufficient certificates, but checking them still requires evaluating the actual likelihood.
-
A value of the entropy reduction ratio near zero does not establish good fit. For driftless diffusion, reversing the sign of the true score leaves the statistic unchanged while changing the reverse dynamics. Matching the integrated score norm is therefore insufficient to support its interpretation as a measure of model fit.
-
The proposed model-selection use is not adequately validated. The width sweeps do not separate the effects of capacity and dissipation, and the natural-image diagnostic depends on an estimated entropy reference. The results do not establish that its plateau reliably identifies where further increases in model size stop improving likelihood or sample quality.
Questions for authors
-
Are there useful conditions under which the residual connecting the surrogate to actual likelihood can be bounded without first evaluating that likelihood?
-
What additional assumptions would make a near-zero entropy reduction ratio informative about model fit despite its insensitivity to score direction?
-
When the true data entropy is unknown, how is the threshold for treating ERR as sufficiently close to zero chosen? How sensitive is the resulting model-size choice to the entropy reference?