acceptodds
Under review as a conference paper at ICLR 2027

Entropy-Based Bounds for Flow and Diffusion Models via Stochastic Thermodynamics

Abstract

We derive a lower bound on the negative log-likelihood of diffusion and flow models from the entropy rates of their noising processes. Our main theoretical result is an exact decomposition of a plug-in surrogate of this quantity into data and noise entropies, a model-induced entropy rate integral, and a nonnegative score-error term. This yields a thermodynamically interpretable lower bound on the surrogate that complements variational upper bounds. The same decomposition gives a model-based upper bound on the differential entropy of the data, and both bounds are tight in the exact-score limit. We validate these results on Swiss roll data and finite Ising systems, where independent entropy estimates or analytic targets are available, and evaluate pretrained image models, including convolutional and transformer score-based diffusion models and a rectified flow, on CIFAR-10, ImageNet-32, FFHQ, CelebA-HQ, and LSUN. Improved likelihood is consistently accompanied by increased integrated entropy production, revealing an accuracy–dissipation relation within variance-preserving and variance-exploding model families. We also introduce the entropy reduction ratio, a likelihood-free measure of fit computed from the score alone. Together, these results establish entropy rates as both theoretical limits and empirical diagnostics for diffusion and flow models, linking likelihood, data entropy, and irreversibility within a unified stochastic thermodynamic framework.

Primary area:
generative models
Keywords:
score-based generative models, diffusion models, flow matching, stochastic thermodynamics, entropy production, negative log-likelihood, entropy rates, probability flow ODE, denoising score matching, differential entropy estimation, nonequilibrium statistical mechanics

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_xV3cOpenReview ↗
Rating
2
Confidence
2
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper relates entropy changes in diffusion processes to the scores learned by generative models. It derives an exact decomposition of a likelihood surrogate into entropy terms and an integrated squared score error, yielding a lower bound on the surrogate and an upper bound on data entropy. A separate residual describes the difference between the surrogate and probability-flow likelihood. Experiments cover synthetic distributions, finite Ising systems, and pretrained image models. The paper also introduces the entropy reduction ratio as a diagnostic and explores its use for choosing model size.

Critical strengths
  1. The decomposition gives a clear interpretation of the bound’s slack as integrated squared score error. Deriving the data-entropy bound from the same identity provides a useful connection between score approximation and entropy estimation.

  2. The synthetic and finite Ising experiments allow meaningful comparisons with independent entropy estimates or analytic targets. Measuring the difference between the surrogate and actual likelihood is also valuable, since it makes the limitations of transferring the bound to actual likelihood visible.

Critical weaknesses
  1. The main theorem bounds a surrogate, not the model’s actual negative log-likelihood. Because the residual connecting the two has no fixed sign, the theorem does not establish the general likelihood limit suggested in the introduction. Appendix C.4 provides sufficient certificates, but checking them still requires evaluating the actual likelihood.

  2. A value of the entropy reduction ratio near zero does not establish good fit. For driftless diffusion, reversing the sign of the true score leaves the statistic unchanged while changing the reverse dynamics. Matching the integrated score norm is therefore insufficient to support its interpretation as a measure of model fit.

  3. The proposed model-selection use is not adequately validated. The width sweeps do not separate the effects of capacity and dissipation, and the natural-image diagnostic depends on an estimated entropy reference. The results do not establish that its plateau reliably identifies where further increases in model size stop improving likelihood or sample quality.

Questions for authors
  1. Are there useful conditions under which the residual connecting the surrogate to actual likelihood can be bounded without first evaluating that likelihood?

  2. What additional assumptions would make a near-zero entropy reduction ratio informative about model fit despite its insensitivity to score direction?

  3. When the true data entropy is unknown, how is the threshold for treating ERR as sufficiently close to zero chosen? How sensitive is the resulting model-size choice to the entropy reference?

Reviewer_MFh2OpenReview ↗
Rating
3
Confidence
1
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

The paper derives a lower bound for the negative log-likelihood for diffusion and flow models from the entropy rates of their noising processes. Through experiments, the authors demonstrate that the theoretical bound holds for toy datasets where a ground truth exists, as well as for high-dimensional datasets.

Critical strengths

The paper is well written. The empirical results provide sufficient validation for the derived theory.

Critical weaknesses

I am not familiar with the entropy calculations for diffusion and flow models. The paper seems to propose novel results that agree with theory, but I do not see any comparison with previously proposed NLL lower bounds.

Questions for authors

None

Reviewer_8XFpOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

The paper aims to derive a lower bound on negative loglikelihood (NLL) of diffusion models. The bound was derived using entropy rate decomposition of a plugin surrogate of NLL. This is then applied to estimate the data entropy from a trained score predictor. Several experiments on different datasets like CIFAR10, CelebA, LSUN were conducted, toy models like swiss roll and Ising were also used as demonstration. The authors attempted to measure the data entropy as well as the newly introduced entropy reduction ratio the show the validity of the lower bound.

Critical strengths
  • I think the experiments are comprehensive, and is a good mix of toy models with analytical solutions, as well as commonly used datasets. The reviewer recognizes that the amount of experiments/computation done is quite impressive, for e.g. line 982-983.
Critical weaknesses
  • The main result (Theorem 1 and its Corollary) is not novel, or at least does not deserve to be a theorem. Most, if not all, of the ingredients of Theorem 1 (first claimed contribution) already exists in Song et al. 2021a (arXiv:2101.09258). For example, Eq. 14 (line 896) is exactly the same as Theorem 4 from Song's, with only some name changes. Another example is the data entropy estimate (second claimed contribution) Eq. 11 also existed as a remark in Song's Theorem 4 and Theorem 5 (Eq. 30). Next, Corollary 1 is also very similar to Song's Theorem 5 eq. 32. Lastly, the ERR is also simply the normalized version of Song's data entropy estimate (Theorem 4). While the reviewer recognizes that these results can be derived independently since the steps are all standard, and the NLL_plug object seems to be new, it still does not merit a full theorem/contribution as presented in the submission.
  • Another main claim at line 058 seems misleading. It's based on Corollary 1, but the authors actually commented that NLL_plug is not actual NLL (line 192-193) as well as inferred from the experiment results on Ising's model (line 461-464). Moreover, the object RθR_{\theta} is studied in Lu et al. (arXiv:2206.08265), which is not cited.
Questions for authors
  • Please clarify the differences/novelty of the main results. Why does it merit a dedicated Theorem?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.