acceptodds
Under review as a conference paper at ICLR 2027

The Loss Floor of Denoising Score Matching: Fisher Geometry from Schr\"odinger Bridges

Abstract

Denoising score matching regresses onto a conditional score, while the generative dynamics require the marginal score. The two objectives share a population minimizer, but at fixed noisy state the conditional target is still random, and this inflates the training loss by a term no model can reduce. We show that this excess is exactly the trace of the Fisher–Rao metric of the conditional endpoint family, integrated along the diffusion, for any corruption kernel meeting mild regularity conditions. Deriving it from the Schr\"odinger bridge variational principle makes its origin precise, the first variation of the bridge functional gives the generative drift, the second gives the metric, so the information geometry observed in diffusion latent spaces is an intrinsic part of the training loss rather than a structure imposed on the model. For corruption diffusions the floor factorizes into a data-dependent information flow and a schedule-dependent weight, and for Gaussian corruption it evaluates to the mutual information the noisy state loses about the data. It is also measurable where, one extra backward pass recovers it from a trained network, and on CIFAR-10 it is of the reported loss on the noise range in standard use, large enough that two checkpoints of a single run are ranked backwards by a factor of , an inversion that subtracting the floor repairs. The same decomposition applies to masked diffusion, where the floor is the entropy of the data and is likewise schedule-independent.

Primary area:
generative models
Keywords:
diffusion, Schrodinger bridge, Fisher-Rao metric, mutual information

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_XiztOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

The paper studies the constant gap between the denoising and ideal score-matching losses. It shows this gap is a Fisher-metric trace, which for Gaussian noise equals the mutual information lost across the SNR range. It estimates the gap from a trained network. On CIFAR-10 the gap is up to 99.86% of the loss, and raw losses misrank models across noise ranges until it is subtracted.

Critical strengths

None. I learn nothing.

Critical weaknesses

First, the studied problem lacks significance. The floor is a model-independent constant with zero gradient, and thus it affects neither training nor sampling. Its only use, correcting cross-configuration loss comparisons, addresses a practice the paper does not show exists. The paper’s own recommendation of common-range evaluation already avoids that issue.

Second, the analysis offers nothing new. The constant follows from the equivalence of explicit and denoising score matching in Vincent (2011). The subsequent results apply the de Bruijn identity, the I–MMSE relation, and the schedule invariance already known from VDM. The Schrödinger-bridge and Fisher-geometry framing is not used by any result.

Third, the experiments yield no useful insight. They show that the constant is large and that subtracting it reorders a constructed cross-range comparison. The resulting excess is validated only against training steps, never against sample quality or likelihood.

Questions for authors

No more questions

1 comment

Thank you for the review. We have withdrawn the 99.86% floor estimate and the claim that floor subtraction repairs rankings across different noise ranges. Below we address each concern and identify the changes in the revised manuscript.

1. Significance and the zero-gradient objection.

We agree: subtracting a model-independent constant changes neither training gradients nor the sampling dynamics of a fixed model. Ordinary DSM loss already ranks predictors on a common objective. Changing the noise interval or weighting changes both the floor and the ideal loss, so subtraction does not make different objectives comparable.

The revised practical question is how much of one predictor's expected denoising loss remains reducible. For network risk LA(f)L_A(f), Bayes floor F(A)F(A), and any reference predictor gg using the available inputs,

F(A)≤LA(g),LA(f)−F(A)≥LA(f)−LA(g).F(A)\leq L_A(g),\qquad L_A(f)-F(A)\geq L_A(f)-L_A(g).

A better reference therefore certifies attainable improvement without estimating the floor accurately. This motivates the paired reference audit, rather than a correction to optimization or model ranking.

Manuscript: Introduction, pp. 1-2; Section 5, Eq. (5.2) and Table 1, pp. 7-9; limitations, p. 10.

2. Novelty, prior identities, and the role of the bridge.

We explicitly credit Vincent's DSM equivalence for the additive constant, de Bruijn for information flow, Guo-Shamai-Verdu for the Gaussian I-MMSE identity, and VDM for weighting invariance. We do not claim these ingredients as new.

The bridge motivates the construction: its first variation gives the generative drift, and the endpoint-tilt family's log-partition Hessian gives the endpoint covariance form. Conditioning the endpoints on the noisy state defines a distinct posterior family. The Markov property splits its Fisher form into past and future blocks; DSM measures the spatial trace of the clean-endpoint block. We distinguish the two families and remove unsupported claims of an isometry or a unique minimal reduction.

Our novelty claim is restricted to this conditional endpoint construction and its connection to DSM target variance. The empirical audit tests whether learned predictors recover that variance. We also credit VDM's discrete decoder and Fourier features; adding those features to the audited networks was not tested.

Manuscript: Sections 2-3, Proposition 3.1 and Theorem 3.2, pp. 2-5; Section 4 and Appendix C; Section 6, scope, p. 9.

3. Experimental evidence and downstream relevance.

We replaced the ranking-reversal interpretation and training-step proxy with paired held-out comparisons. On CIFAR-10 over log-SNR [13,13.8], the 240k-step network has risk 1095.0 +/- 1.9 nats/image. Hard rounding and the soft discrete reference reduce paired risk by 994.5 +/- 2.0 and 1017.9 +/- 1.9 nats/image, respectively. These comparisons measure improvements on an identical objective.

An independent audit assigns 1100.2 nats/image to the learned Jacobian estimate, while the pixel-lattice bound puts the true floor at most at 100.6. The Jacobian identity is exact for the Bayes predictor; a learned predictor can bias it. Agreement with the forward-only Stein expression checks the same plug-in quantity and does not remove this bias. The soft reference uses known pixel support without assuming the true image posterior or diagnosing a unique cause of the network's error.

A frozen class-conditional ImageNet-64 EDM denoiser provides a second dataset and architecture: the soft reference reduces risk by 6.9% on the tested band. Reported uncertainties are Monte Carlo standard errors. The CIFAR checkpoints share one training trajectory; sample quality, likelihood, and model selection by those criteria remain untested. Appendix E now explains the predictors, evaluation protocol, and uncertainty.

Manuscript: Sections 5.1-5.3, Eq. (5.1), Figure 1 and Table 1, pp. 7-9; Appendix E, pp. 25-27; limitations, p. 10. Appendix D.5 now uses the analytic range example only to explain changes in the objective.

Reviewer_HggdOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

The paper studies the gap between the denoising score-matching objective and the marginal-score objective. It identifies this gap with the integrated trace of the Fisher–Rao metric of the posterior over clean endpoints, connects it to a Schrödinger bridge formulation, and derives information-theoretic expressions. It also proposes a network-Jacobian estimator of the floor and reports CIFAR-10 experiments in which the estimated floor accounts for most of the measured loss. The main practical claim is that removing this floor improves the interpretation and comparison of reported diffusion losses.

Critical strengths
  • The connection between conditional-target uncertainty, posterior Fisher geometry, and information flow is an interesting theoretical perspective on the DSM objective.
  • The paper goes beyond identifying an additive constant by proposing an estimator and investigating its magnitude on CIFAR-10.
Critical weaknesses

1. The paper is difficult to follow

  • Clarity is my main concern. I found the arguments in Sections 2 and 3 difficult to follow because several equations needed to understand the transitions are left to the appendices. The main text compresses the passage from the bridge variational problem to the score-matching objective into verbal statements whose assumptions and intermediate steps are not sufficiently explicit. This makes it difficult to assess both the derivation and its purpose: what is the point of using the bridge formulation when the conditional DSM loss can be derived directly ?

  • Several notations are also not defined. The candidate path measure H\mathcal H appears in the Sanov expression without a clear introduction, and the backward field ψ\psi is introduced without its defining equation or terminal condition. In Section 3.1, the endpoint potentials u,vu,v, the family H(u,v)\mathcal H^{(u,v)}, and the log-partition functional need explicit definitions and a connection to the earlier factors f,gf,g. Reusing vv for both a reference drift and an endpoint potential adds ambiguity.

  • The central identity is stated and proved in Section 3.2. I would suggest presenting this argument earlier and then explaining what the bridge and Fisher–Rao interpretations add. The reader should not have to reconstruct the appendix derivations to understand the paper's main message.

2. The practical consequences of the formulas are not established

  • The practical claim is unclear. Across different schedules or weightings, both the floor and the ideal score-matching objective change. Removing the floor therefore does not generally produce a common measure of model quality. The paper needs to identify what floor estimation adds beyond evaluating all models under one common objective. The claimed 5.9-fold ranking inversion illustrates this concern: the early checkpoint is evaluated on a narrow noise range and the later checkpoint on a wide range. Subtraction reverses their numerical ordering, but the residuals still measure different ideal objectives.

  • I like the MMSE representation in Section 4.2 and the decomposition in Equation (3.6), but the paper does not develop their consequences enough to make the significance convincing. What does the MMSE curve reveal about different data distributions and noise regimes? The paper provides several interpretations of the same quantity, but leaves their added explanatory value insufficiently developed.

3. There are other changes to the original KL objective.

Section 2 passes from the original path-space KL to DSM through several logically different operations, in particular (i) subtracting the optimal bridge cost using the Pythagorean identity and (ii) replacing the marginal score with a conditional score. The subsequent analysis focuses on the last operation, while the Pythagorean decomposition already changed the objective by a model-independent constant i.e. also changes the objective. Studying the DSM offset then needs a clearer motivation than the fact that it changes the objective's value.

Appendices A.4 and A.5 need more specific attribution if they reproduce existing background.

Questions for authors

See above

1 comment

Thank you for the detailed comments. We revised the derivation and interpretation while retaining the bridge motivation. Locations below refer to the revised manuscript.

1(a). Missing steps from the bridge to DSM.

Section 2 defines the sigma-finite reference, endpoint-constrained probability laws, and factorization H∗=fGg\mathcal H^*=f\mathcal G g. It gives the backward equation with potential, terminal condition ψT=g\psi_T=g, Doob drift, and conditional-Jensen derivation of DSM. Identifying ψ=Pt\psi=P_t requires a unit forward factor and compatible initial law. Appendix A states entropy/Girsanov conditions, the prior correction, and positive-noise scope for atomic data. Finite reference normalization adds a constant; Sanov applies to probability references.

Manuscript: Section 2, Eqs. (2.1)-(2.3), pp. 2-3; Appendix A.1-A.5, pp. 15-19, including the Kramers-Moyal expansion in A.2.

1(b). Potentials, log-partition, and notation.

We define scalar potentials u,vu,v, joint endpoint law ρ\rho, and log-partition Λ\Lambda before the covariance calculation. Equation (B.1) defines Λ\Lambda explicitly. We distinguish scalar vv from the drift, the full metric gendg^{\rm end} from its past/future blocks, and the clean-endpoint spatial metric gg used in the loss. Gaussianity and isotropy are stated assumptions on the corruption.

Manuscript: Section 3.1, Proposition 3.1, pp. 4-5; Appendix B.1-B.2 and B.4-B.6.

1(c). State the central identity earlier; explain the bridge.

The Introduction gives Eq. (1.1) and explains the vanishing cross term. Section 3.2 proves it by conditional expectation, with regularity and finiteness conditions. The bridge motivates the dynamics and endpoint family; the elementary regression identity does not require it. The endpoint geometry identifies which Fisher block the loss measures.

Manuscript: Introduction, p. 2; Sections 2-3, especially Theorem 3.2 and Eqs. (3.3)-(3.6), p. 5.

2(a). Floor subtraction cannot compare different objectives.

We agree and withdraw the ranking correction and 99.86% floor estimate. Both loss components change with interval and weighting. Equation (5.2) instead bounds reducible error through a lower-loss reference on the same objective. Paired CIFAR-10 improvements and an independent lattice bound replace the original interpretation; neither requires a Bayes-optimal reference or a tiny loss-minus-floor subtraction.

Manuscript: Introduction, p. 2; Section 5, Eq. (5.2), Table 1; Appendix D.5; limitations, p. 10.

2(b). What the curves and local decomposition explain.

For Gaussian variance aa, MMSE is a/(1+snr a)a/(1+\mathrm{snr}\,a) and the contribution per unit log-SNR is M=snr MMSEM=\mathrm{snr}\,\mathrm{MMSE}. The scale snr a=1\mathrm{snr}\,a=1 marks resolution of that covariance direction. At high SNR, MMSE vanishes but MM tends to one; for finite alphabets, MM vanishes and information saturates at the data entropy. This corrects the unqualified divergence claim. The posterior Fisher trace is conditional target variance. We define translation Fisher information and distinguish posterior radial KL from marginal-averaged information flow.

Manuscript: Sections 3.2 and 4.1-4.2, pp. 5-7; Appendix B.7, p. 22; Appendix D.7, Eq. (D.1), p. 25.

3(a). Bridge-cost subtraction versus the DSM offset.

We separate three operations. First, the Pythagorean subtraction gives $D_{KL}(\mathcal{H} | \mathcal{H}^{\star}) $ only when both endpoint constraints hold; otherwise there is an endpoint correction. Second, Girsanov averages drift error under the candidate law. Replacing this by fixed-data averaging defines a different risk and can change the best restricted-model approximation. For the reverse-corruption specialization with matching weight and initial law, the ideal score risk equals DKL(H∗∥H)D_{KL}(\mathcal H^*\|\mathcal H); an approximate prior adds initial-law KL. Third, replacing the marginal-score target by a conditional target adds posterior variance. This DSM offset, distinct from the optimal bridge cost, is the term analyzed geometrically and empirically.

Manuscript: Section 2, p. 3; Section 3.2, p. 5; Appendix A.5, Eqs. (A.34)-(A.35), p. 19.

3(b). Attribution for Appendices A.4-A.5.

We credit Premkumar for the action-principle derivation, Vincent/Song-Ermon/Ho for DSM, and Leonard/Song et al. for path-KL and Girsanov. We distinguish the exact frozen-Gaussian calculation from the continuous identity and remove the unsupported accumulated-discretization-rate claim. Novelty is restricted to the conditional endpoint construction and its connection to DSM variance.

Manuscript: Section 2; Appendices A.4-A.5, pp. 17-19; Section 6, scope, p. 9.

Reviewer_VTbyOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper derives the generative drift of diffusion models from a Schrödinger-bridge variational formulation and links the geometry of the corresponding endpoint distributions to the denoising training objective. Its central aim is to characterize the irreducible loss introduced when replacing the intractable marginal score with a conditional-score training target.

The authors prove that the denoising objective equals the ideal marginal-score objective plus a model-independent loss floor. This floor is precisely the integrated trace of the Fisher–Rao metric of the posterior distribution over clean data conditioned on the noisy state. For corruption diffusions, it factorizes into a schedule-dependent weight and a data-dependent information flow; under matching loss weighting, it equals the mutual information lost during corruption.

The paper further estimates this floor on CIFAR-10 and shows that it can dominate the reported loss, making raw losses misleading when comparing across different noise ranges or weighting configurations.

Critical strengths

The paper clearly introduces its central problem and contributions, providing a useful roadmap through the theoretical development. The distinction between the ideal marginal-score objective and the tractable denoising objective is well-motivated, and Section 2 clearly explains the irreducible gap introduced by conditional regression.

The connection between the loss floor, Fisher–Rao geometry, and information flow provides a coherent theoretical perspective. Section 5 complements this analysis with a practical estimator and CIFAR-10 experiments, illustrating how the floor can dominate reported losses and distort comparisons across noise ranges.

Critical weaknesses

Section 3 is particularly notation-dense, and several transitions would benefit from additional explanation. More explicit definitions and intuitive guidance on the relationships between the path-space formulation, endpoint distributions, and the latent metric would improve accessibility without sacrificing rigor.

The empirical validation is limited to CIFAR-10 with a single training run per noise range. Although this does not undermine the theoretical identities, it leaves the estimator’s reliability and practical usefulness across datasets, architectures, and training seeds insufficiently established. Furthermore, the experiments do not demonstrate whether floor-corrected loss improves model selection according to generated sample quality.

Questions for authors

a. How does the irreducible loss floor vary across different noise schedulers? In particular, under what conditions is changing the schedule merely a time reparameterization that preserves the floor, versus when it changes the effective loss weighting or SNR endpoints and therefore alters the floor?

b. I would like more insight into how the floor's magnitude and weighting adapt across different instances of score-matching models, including flows, rectified flows, and continuous normalizing flows. Is there a common definition, and do some of these framework variants provide a better approximation of the gap?

c. What is the effect of learning from noisy data within the currently proposed formalization? Would this increase the gap, and is there a way to relate this to the score-matching objective?

d. Can you replicate the results beyond the CIFAR-10 dataset on more complex and intertwined data distributions?

e. When the floor accounts for 99.86% of the loss, how reliably can subtraction resolve the small, model-dependent remainder? Could you elaborate on the potential numerical precision or instability issues associated with this step?

2 comments

Thank you for the review. We address both weaknesses and questions (a)-(e), with revised-manuscript locations.

Weakness 1: clarity and notation.

We clarify the sequence: bridge variation gives dynamics and an endpoint-tilt family; its log-partition Hessian gives endpoint covariance; conditioning endpoints on the noisy state gives a distinct family whose clean-endpoint spatial Fisher trace is DSM target variance. We define potentials and endpoint laws, distinguish drift and metric blocks, and state Gaussian assumptions and time/SNR conventions. Appendix B.3 explains exact path-conditioning singularity with references.

Manuscript: Sections 2-3, Proposition 3.1 and Theorem 3.2, pp. 2-5; Appendix B.

Weakness 2: empirical scope and model selection.

We add the ImageNet-64 audit in (d). CIFAR checkpoints share one training run; repeated evaluations measure conditional sampling variability. Independent training seeds, sample quality, likelihood gains, and broader estimator reliability remain untested. We withdraw the cross-objective ranking correction.

Manuscript: Sections 5.2-5.3, pp. 8-9; Appendix E; limitations, p. 10.

(a) Schedule invariance.

For fixed affine Gaussian corruption, λ=log⁡snr\lambda=\log\mathrm{snr} and F=12∫λ0λ1ωM dλF=\frac12\int_{\lambda_0}^{\lambda_1}\omega M\,d\lambda, with M=snr MMSEM=\mathrm{snr}\,\mathrm{MMSE} and ω=w/(σ2∣λ˙∣)\omega=w/(\sigma^2|\dot\lambda|) for score-target weight ww. For noise prediction, ω=w/∣λ˙∣\omega=w/|\dot\lambda|. Invariance requires the same corruption family, SNR endpoints, and induced weight. For increasing t=ϕ(s)t=\phi(s), the weight becomes w(ϕ(s))ϕ′(s)w(\phi(s))\phi'(s); matching evidence-bound weighting gives ω=1\omega=1. Changed endpoints or uncompensated sampling density change the integral. VP/VE rescaling needs compatible targets and weights. Quadrature, sampling variance, and optimization can still change.

Manuscript: Section 4.2, Eqs. (4.3)-(4.4); Lemma C.1; Appendix C.1, p. 23.

(b) Flow matching, rectified flows, and CNFs.

Squared-regression floors are weighted conditional target variances, conditioning on all model inputs. For xt=αty+σtzx_t=\alpha_ty+\sigma_tz, independent Gaussian zz, and T=aty+btzT=a_ty+b_tz, target covariance is ct2Cov⁡(y∣xt)c_t^2\operatorname{Cov}(y\mid x_t), with ct=at−btαt/σtc_t=a_t-b_t\alpha_t/\sigma_t. Table 2 gives score/noise/data/velocity coefficients. Flow velocity includes a time-scale factor: equal weights for trigonometric VP give π2/4\pi^2/4 relative to vv-prediction. Endpoint coupling can change the posterior covariance itself. The regression identity covers flow-matching-trained CNFs, not automatically likelihood objectives. We make no accuracy ranking of these frameworks.

Manuscript: Section 6; Appendix F.1-F.3, Proposition F.1 and Table 2, pp. 27-29. F.4/Table 3 are identity checks, not trained-model comparisons.

(c) Noisy training observations.

DSM learns the further-corrupted observed-data score; floor subtraction does not recover the clean law. For independent additive observation noise and fixed Gaussian training corruption, weight, and interval, convolution cannot increase translation Fisher information, giving Fobs≥FcleanF_{\rm obs}\geq F_{\rm clean}. Appendix C.2 proves this with a Gaussian example, excluding arbitrary observation channels or changed weights. Continuous observation noise also changes the lattice-bound support assumption.

Manuscript: Section 6, p. 10; Appendix C.2, pp. 23-24.

(d) Replication beyond CIFAR-10.

For frozen class-conditional EDM on 512 ImageNet-64 validation images, over σ∈[0.002,0.003]\sigma\in[0.002,0.003], network risk is 0.3494 nats/dimension and soft-reference risk is 0.3252. The paired reduction is 0.0242 +/- 0.0022, or 6.9%; hard rounding is worse at 0.4380. A class-ignoring reference still bounds the class-conditioned floor. We integrate each image's paired differences across nine noise nodes before computing standard errors, retaining cross-node covariance. The improvement concerns this band, not the full range.

Manuscript: Section 5.3, p. 9; Appendix E, protocol and E.1, p. 27.

(e) Resolving a 0.14% remainder.

We withdraw the 99.86% estimate and do not claim to resolve that remainder. Appendix E.1 separates learned-predictor bias, sampling error, and floating-point error. Pairing accounts for covariance in Var⁡(L^−F^)\operatorname{Var}(\widehat L-\widehat F), but small standard errors, more probes, or higher precision do not remove plug-in bias. The appendix documents FP32/FP64 evaluation and the limits of later precision conversion. We instead use paired reference improvements to lower-bound reducible risk. Standard errors exclude plug-in bias, training-seed variation, and quadrature error.

Manuscript: Section 5, Eq. (5.2); Appendix E.1, Eq. (E.2), p. 27; limitations, p. 10.

I thank the authors for the clarifications, but I will keep my score.

Comments on the paper

3 comments

Thank you for the reviews. Below we address each referee's weaknesses and questions in report order. The concern headings are paraphrases; all section, equation, figure, table, and page references refer to the revised manuscript.

One correction applies to all three reports: we withdraw the original 99.86% floor estimate and the claim that floor subtraction repairs model-quality rankings across different noise ranges. The revised experiments test a learned floor estimate against independent bounds and compare predictors on the same denoising objective. The manuscript retains the Schrödinger bridge as the starting construction, while making the assumptions and the separate conditional-regression argument explicit.

Thank you for clarifying the derivation and withdrawing the cross-range ranking claim. I still insist on two concerns I had:

  1. Role of the Schrödinger bridge. Vincent’s DSM equivalence gives the additive gap between the conditional-score and marginal-score objectives. Since the marginal score is the posterior mean of the conditional score, that gap is the conditional variance of the denoising target. By Bayes’ rule, we can recover the Fisher trace, and integrating over noise levels yields Theorem 3.2. Which conclusion about the loss floor depends on the bridge formulation from section 2 ?

  2. Practical significance of the revised experiment. The new CIFAR-10 experiment shows that the learned Jacobian estimate can substantially overestimate the floor. I see this as a concrete failure case for the proposed estimator. With the ranking claim withdrawn, it remains unclear what benefits from your floor estimation.

Dear Reviewers,

Thank you for your time and thoughtful reviews. With the author–reviewer discussion period ending on November 18 (AoE), please review the authors' rebuttal and share your updated assessment. Specifically, let us know whether your main concerns have been resolved, and if not, which points remain open.

Your input before the deadline is greatly appreciated!

AC

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.