acceptodds
Under review as a conference paper at ICLR 2027

Conditioning Degenerate Diffusion Models

Abstract

Current conditioned generative models heavily rely on score functions for guidance during training. When the generative model is a diffusion process with a singular diffusion coefficient and the underlying (conditional) densities either do not exist or are not smooth, we use causal optimal transport to define approximate loss functions that identify a minimum-entropy control for guidance under minimal assumptions. Our approach relies on causal optimal transport and its characterization through the predictable representation property of (conditioned) diffusion processes whose associated martingale problem is well posed, \`a la \"Ust\"unel.

Primary area:
generative models
Keywords:
generative ai, diffusion models, score-based models, conditioning diffusion models

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_nVh8OpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper develops regression losses for conditioning diffusions with rank-deficient diffusion coefficients, including settings without smooth ambient-space densities. Using predictable representation in the state filtration, it identifies the minimum-energy Brownian control by regressing short projected or raw Brownian increments. A state-increment formulation recovers the induced drift and, via the pseudoinverse, the corresponding control. The analysis covers random initialization and density processes that may vanish, and establishes population convergence on compact intervals before the terminal time under the stated well-posedness, local absolute-continuity, square-integrability, and feedback assumptions. The paper also presents an endpoint-control formulation and illustrates the method on a nonlinear diffusion with a numerical Doob-bridge reference and a Fashion-MNIST inpainting proof of concept on a fixed affine subspace.

Critical strengths
  1. A regression formulation that accommodates degenerate noise. Theorems 1–2 and Corollary 1 turn predictable representation into regression targets that require neither an ambient-space score nor reconstruction of the entire driving Brownian motion. They explain which noise directions affect the state and how the raw-increment formulation avoids numerical projection. This broadens the setting relative to the Euclidean derivation using invertible volatility in Pidstrigach et al. (2025).
  2. Careful treatment of the assumptions behind the main results. Appendix C handles random initialization and changes of completed filtrations, and the localization argument accommodates density processes that may vanish. Theorem 7 accounts for initial entropy. The paper distinguishes existence of a controlled solution under a changed measure from weak uniqueness of independently simulated solutions. Appendix G also distinguishes convergence for individual conditions from pooled convergence and treats discretization of the projected and raw targets separately.
Critical weaknesses
  1. The image experiment needs a clearer link to the regression theory. Section 5.3 trains under the empirical bridge-mixture law RνTR_{\nu_T} using a finite-window increment. Theorem 11 establishes an exact identity under the additive reference law. Specifying how the implemented increments are constructed and deriving their conditional expectation under the actual training law would clarify which drift the regression identifies.
  2. The numerical evidence supports a narrower set of claims.Figure 1 gives a qualitative comparison with the numerical Doob bridge, but leaves the accuracy of the three increment windows unquantified. Figure 2 illustrates inpainting feasibility. Its stronger claims about conditional scores and time reversal need a separate argument in the 648 accessible coordinates, since the absence of an ambient-space density alone does not establish an obstruction on the accessible support.
Questions for authors

ould you clarify which admissibility conditions justify the entropy identity used for arbitrary competitors in the proof of Theorem 10? Theorem 7 explicitly includes its corresponding identity in the definition of admissibility. Does Theorem 10 require a similar condition? Please also clarify whether the uniqueness claim concerns the control, the state-path law, or the pair (Q,v)(Q,v). 2. How are the increments in Section 5.3 constructed under RνTR_{\nu_T}, and what is their conditional expectation given (t,Xt,Y)(t,X_t,Y)? Does the Gaussian bridge calculation give exact population identification for the implemented finite window? Please distinguish identification under the empirical mixture from generalization to held-out conditions. 3. Which obstruction to conditional scores or time reversal remains in the 648 accessible coordinates? Please distinguish ambient-space densities from densities on the accessible support. 4. Could you quantify the discrepancy from the numerical Doob bridge for the three increment windows, for example using effective-drift or endpoint error, and report the simulation time step and terminal cutoff? 5. Could the abstract replace “à la Üstünel” with a direct explanation of how predictable representation leads to the regression targets and specify the assumptions behind the population result? Please distinguish these from the additional assumptions used for discretization and learning in Appendix G. Section 2.1 also appears to omit the conditioning when it states that the original and controlled processes have the same law. 6. Could you label the ground truth, observed condition, and generated completion directly in Figure 2? If the figure is intended to support a comparative advantage, what evidence supports that claim?

2 comments

We thank to the reviewer for pointing out several missing details from the text and explicitly pointing them out. We believe we have addressed most of these concerns, and they can be seen on the updated version of the manuscript, marked with blue. Below, we point out how the issues brought up are addressed by referring to each one separately.

Clarification on Theorem 1010

We have added the missing necessary finite entropy condition for the ``admissibility'' in Theorem 10 so that the relative entropy is well defined. The uniqueness claim concerns the pair (Q,v)(Q,v). This then implies the uniqueness of the control and the induced state-path law. We have highlighted the uniqueness of (Q,v)(Q,v) in the statement and added a remark. The proof remains the same.

How are the increments in Section 5.3 constructed under RνTR_{\nu_T}, and what is their conditional expectation given (t,Xt,Y)(t,X_t,Y)?

We first sample a terminal target Z∼νTZ\sim\nu_T and set Y=G(Z)Y=G(Z). Writing L=ASL=AS, we jointly sample BtB_t and Bt+εB_{t+\varepsilon} from the Gaussian bridge ending at L†(Z−x0)L^\dagger(Z-x_0).

We use the empirical estimator ERνT[Bt+ε−Btε∣t,Xt,Y]\mathbb{E}^{R_{\nu_T}}[\tfrac{B_{t+\varepsilon}-B_t}{\varepsilon}\mid t,X_t,Y], which satisfies $\mathbb{E}^{R_{\nu_T}}[\tfrac{B_{t+\varepsilon}-B_t}{\varepsilon}\mid t,X_t,Y] =\tfrac{\mathbb{E}^{R_{\nu_T}}[B_T-B_t\mid t,X_t,Y]}{T-t}$.

Please distinguish identification under the empirical mixture from generalization to held-out conditions

We included an explain under Figure 22.

Specifying how the implemented increments are constructed and deriving their conditional expectation under the actual training law would clarify which drift the regression identifies.

We have included a further discussion related to this implementation.

Does the Gaussian bridge calculation give exact population identification for the implemented finite window?

It does. This is a special case of Theorem~1111 in the Appendix, so no ε→0\varepsilon \to 0 is required.

Please distinguish identification under the empirical mixture from generalization to held-out conditions.

We have highlighted this in a sentence.

Q3. Which obstruction to conditional scores or time reversal remains in the 648 accessible coordinates? Please distinguish ambient-space densities from densities on the accessible support.

The main ones are that, ∇xlog⁡p(Y=y∣Xt=x)\nabla_x \log p(Y=y|X_t=x) is still singular at t=Tt=T even under the 648 accessible coordinates, which is required to be well defined for the usual time-inversion. Furthermore, the usual conditional score method to learn ∇xlog⁡p(Y=y∣Xt=x)\nabla_x \log p(Y=y|X_t=x) is to learn both ∇xlog⁡pt(x∣Y=y)\nabla_x \log p_t(x|Y=y) and ∇xlog⁡pt(x)\nabla _x \log p_t(x); however, since the initial condition is deterministic and the SDE is on 648648 dimension, at t=0t=0 we have singularity for ∇xlog⁡pt(x)\nabla_x \log p_t(x), which is the marginal distribution of XX at t=0t=0, since the initial condition we have taken is deterministic.

There is no direct obstruction to conditional score under the 648 accessible coordinates in the region 0<t<T0<t<T (however, since this is an open interval there will be large error). However, for most applications we do not know the actual accessible dimensions (and computing them can be expensive). Furthermore, the accessible dimensions might change for all tt with positive probability, so in general one needs to understand all the singularities of the score function. In each of these cases, one needs to pass from the extrinsic space'' (or the ambient space) to an intrinsic manifold to access to the full support. In most applications, this knowledge is not available directly, as the diffusion $X$ merely lies on a noisy manifold'', where the dimension of the support XX can very with positive probability. We refer to

Jakiw Pidstrigach. Score-Based Generative Models Detect Manifolds. NeurIPS 2022,

and references therein for further discussion on this matter. In principle, the singularities of ∇log⁡xlog⁡pt\nabla \log_x \log p_t and ∇xlog⁡p(Y=y∣Xt=x)\nabla_x \log p(Y=y|X_t=x) can also incur in the interior interval (0,T)(0,T). Our aim was to demonstrate that one can do the conditioning without relying on any intrinsic information, which amounts to the 648648 accessible coordinates in that example. We remark that, for generative models, extrinsic structure is always available. We have added further details on this in the caption of Figure 22.

4. Could you quantify the discrepancy from the numerical Doob bridge for the three increment windows, for example using effective-drift or endpoint error, and report the simulation time step and terminal cutoff?

We have added further quantitative results that address this concern in Appendix H.1H.1. Our quantitative results also include a comparison for the different losses we have introduced.

5. Could the abstract replace “à la Üstünel” with a direct explanation of how predictable representation leads to the regression targets and specify the assumptions behind the population result?

We appreciate the suggestions. We believe the second part would be too technical to include in an abstract. We included the first part in the abstract in the updated manuscript.

Please distinguish these from the additional assumptions used for discretization and learning in Appendix G.

We have added a short discussion in the main body after introducing Algorithm~1.

Section 2.1 also appears to omit the conditioning when it states that the original and controlled processes have the same law.

Corrected, thanks.

6. Could you label the ground truth, observed condition, and generated completion directly in Figure 2? If the figure is intended to support a comparative advantage, what evidence supports that claim?

We have added labels to the figure.

The figure was to demonstrate that one can generate these images without relying on local (intrinsic) information, such as local coordinates under which a smooth conditional likelihood (with respect to Lebesgue measure) may or may not exist. We did not aim for any quantitative claim. We have added a brief clarification in the main body related to this. However, in the current version, there are several other numerical examples of the updated manuscript can be found in the Appendix H.

Reviewer_3E2aOpenReview ↗
Rating
3(+1)
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper considers the problem of sampling from conditional SDEs i.e. samples from dXt=b(t,Xt)dt+σu(t,Xt)dt+σ(t,Xt)dWtdX_t = b(t, X_t) dt + \sigma u(t, X_t) dt + \sigma(t, X_t) dW_t Where u is a control term to enforce an end point condition G(XT)=yG(X_T)=y, where GG could be for example ,the identity leading to bridges XT=yX_T=y. Furthermore, the paper considers degenerate SDEs, such as the case that σ\sigma is singular or the density does not exist. To this end, the paper introduces a new form for the control and shows that the control may be written in terms of a projections/pseudo inverses of the trajectories. This is used to learn the control.

Critical strengths
  • I think the ideas in this paper are nice and I learned something from reading it. Specifically, the paper provides a new Tweedie type formula for the control using projections and then, as corollary 2 shows, it can be rewritten in terms of the Brownian motion; I think this is an insightful formula.

  • I appreciate the notation section and think this makes the paper more readable.

  • The paper is formulated without Lebesgue densities, and therefore applies in cases where these do not exist and where the control cannot be written as ∇log⁡h(t,x)\nabla \log h(t, x).

Critical weaknesses

Presentation: I found the presentation made the paper a little hard to follow in places. Although, I appreciate the motivation and explanations throughout Section 4, in places it was hard to understand exactly what they were referring to. For example: Lines 162-165 introduces two independent points but the phrasing makes them seem interrelated and it’s therefore unclear how the first point links to Corollary 1. In line 185-188, I don’t understand how this links to Corollary 2.

Experiments: Although the mathematical contribution is nice, since the contribution is a method I would expect some more experiments. Currently there are no comparisons to existing methods. For example, on the non degenerate case it would be good to compare to Pidstrigach et al. who are cited, as well as a subset of [1, 2, 3, 4] e.g. [1, 2]. In particular, I believe [1, 2] also hold in the case that the SDE is hypoelliptic. In Section 4, different regression targets are also introduced which Remark 4 notes lead to different algorithms. It would also be beneficial to compare these to understand which works best.

Metrics: For the Double-Well experiment, since there is a ground truth it would also be useful to provide metrics. Although Figure 1 is a nice illustration of what the learned paths look like for different ϵ\epsilon, it is hard to see much difference visually, and a quantitative comparison would help.

[1] Yang et al. https://arxiv.org/abs/2502.11909

[2] Howard et al. https://arxiv.org/abs/2512.05070

[3] Heng et al. https://arxiv.org/abs/2111.07243

[4] Baker et al. https://arxiv.org/abs/2407.15455

Minor:

  • In Theorem 3 (informal) ξ\xi is undefined.
  • In line 52 it’s stated that XX and XuX^u have the same law, which is incorrect.

Overall: I think the mathematical contribution of this paper is strong. However, due to the current lack of comparisons in the experiments to other methods and of the regression targets introduced in the paper, I currently score the paper as a weak reject. For this reason I also give a 2 for soundness. Before the paper can be accepted for publication, I think the experiment section needs to be made stronger.

Questions for authors

Can the authors clarify which regression target is expected to work best?

2 comments

We thank the reviewer for their thorough review.

Presentation

We have tried to express ourselves more clearly using the extra pages. Thank you for pointing these issues out.

Experiments

Special cases of the results of Pidstrigach et al. overlap with Theorem 11 in the appendix. We have added comparisons among our increment methods and with other methods in the Appendix H.3. In special cases, we can also use previous score-based methods by using the local geometry. We have compared our methods with DPS (Diffusion Posterior Sampling) in the low-noise case to demonstrate that previous score-based methods can behave badly near singularities of the scores.

One of the properties we wanted to highlight is the importance of closed-loop controls, which are required in cases with hard constraints where understanding the path of the actual process, rather than just its output (as in image generation), is important. We have therefore added pathwise comparisons (when there exists a ground-truth reference). Due to the potential instability of the estimators we have provided, we have decided to include a FBSDE solution using the minimum entropy property of the control (see Section 5.1).

The results of Pidstrigach et al. also work under hypoellipticity with some minor modificaitons. In Appendix H.3 we have also added a comparison of our method with some of the other mentioned results. We have observed that, even when σ\sigma and bb are not constant, the runtime of Pidstrigach et al.'s method is similar to that of ours in that experiment; however, Malliavin calculus based approaches can provide worse pathwise error bounds, even though they can be faster at attaining the desired condition. The results of Yang et al. and Howard et al. were slower.

Minor comments

We have made the corrections.

Q: Can the authors clarify which regression target is expected to work best?

We did not include all the experiments we ran on these regression targets; however, Brownian increments (Corollary 11) usually have the worst pathwise error compared with the references and endpoint conditioning, while having the fastest runtime. Theorem 22, in contrast, is the slowest but yields the lowest error rates. The runtime of the regression target introduced in Theorem 11 is usually in between; however, in some cases, we have experienced numerical instabilities due to the projection term.

Thank you to the authors for their response and the additional experiments. I think the comparison of the regression targets and the comparison to existing methods is useful. However, some details are still missing, which makes the results harder to interpret. For example, unless I missed it, I could not find in the paper which modifications were made to Pidstrigach et al. to apply it in the hypoelliptic setting. For Yang et al., which auxiliary linear process was used for the guided proposal? For Howard et al., why was SC1 used rather than SC2, which they report performs better for double-well problems? If this was due to the hypoellipticity, it would be useful to state this. Were all methods trained with the same network architecture and budget, or with the original authors' code and settings? Furthermore, I think some interpretation or discussion of the results would be valuable, for example why the relative performance of the baselines differs from that reported in Howard et al. and Yang et al., and what the near-identical performance of the three regression targets implies for choosing between them. Including the observations mentioned in the rebuttal would also help.

Reviewer_ennqOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes regression losses for conditioning SDE's with rank-deficient diffusion coefficients. It learns the conditioning control from stochastic increments and extends pre-existing martingale-representation results to random initializations and likelihood ratios that may vanish.

Critical strengths

o. The regression formulation could be useful when a conditional score is unavailable or difficult to evaluate. In particular, the raw Brownian-increment target avoids computing the projection required by the theoretical construction.

o. Authors explicitly address generalizations beyond the smooth Doob transform. The appendix also includes population-convergence results and an analysis of the discretized regression targets in detail.

Critical weaknesses

Novelty

o. The central estimator follows a familiar martingale-conditioning argument. Taking the conditional expectation of a terminal weight, applying martingale representation, and computing its cross-moment with a Brownian increment already gives the basic regression identity. This does not require differentiating a smooth h. The extensions concerning the state filtration, random initialization, and minimum-energy properties need to be distinguished from the estimator itself.

o. Rank deficiency and the absence of a smooth conditional likelihood are different issues. The drift can propagate stochasticity into directions that receive no direct noise. Therefore, a singular diffusion matrix alone does not establish that the usual score-based construction is unavailable. The relevant issue is which additional cases the proposed method handles and why those cases matter.

o. The motivation moves between conditioning a prescribed stochastic process and designing a generative model. These are different problems. In the first case, the dynamics are given. In the second, the authors need to explain what benefit their choice of degenerate dynamics provides over a non-degenerate model in an appropriate representation. The fashion experiment does not establish that benefit.

Efficiency

o. The method learns the conditioning control through stochastic regression, rather than evaluating a derivative of the score. This can avoid an analytic calculation, but it introduces a statistical estimation problem. For the raw Brownian-increment target in Corollary 1, a direct calculation under the reference law gives:

EP∥Bt+ε−Btε∥2=dε.\mathbb E^P\left\|\frac{B_{t+\varepsilon}-B_t}{\varepsilon}\right\|^2=\frac{d}{\varepsilon}.

The target therefore becomes increasingly noisy as the window shrinks. The paper does not establish how many trajectories are needed to estimate its conditional mean accurately, or how that requirement changes with dimension and ϵ\epsilon.

o. Appendix G provides limiting and target-discretization results, but these do not quantify the accuracy–cost trade-off of the trained sampler. Appendix F gives an exact terminal-increment alternative for additive systems, which is useful, but does not resolve the statistical question for the general setting. A quantitative study of estimation error, sample count, increment window, and runtime is missing.

Experiments

o. The nonlinear example does not demonstrate the difficulty motivating the paper. Although the full diffusion matrix is singular, the conditioning variable follows an autonomous one-dimensional diffusion with nonzero noise. The authors themselves compute the reference control using a one-dimensional backward Kolmogorov equation. This verifies a simple instance, but does not demonstrate a case where a smooth conditional likelihood is unavailable.

o. he Fashion-MNIST experiment introduces degeneracy through a known PCA representation. The accessible space has 648 dimensions inside the 784-dimensional image space. Why not perform ordinary diffusion directly in those retained coordinates? This is the natural controlled baseline, but it is not evaluated. The absence of an ambient image-space density is not sufficient to establish a computational obstacle in this example.

o. The numerical evidence is almost entirely qualitative. Figure 1 shows sample paths and empirical means, despite having a reference control available for a quantitative comparison. Figure 2 shows eight image completions without a baseline or an accuracy–cost evaluation.

o. Both examples use deterministic initializations and constant diffusion matrices. Neither demonstrates the practical benefit of several of the generalizations developed in the paper.

Overall

I am not convinced that the paper establishes a sufficiently important contribution to generative modeling. The authors do generalize existing stochastic-analysis results, but the experiments do not demonstrate why that generalization is needed or what quantitative advantage it provides.

The issue is not that a theoretical paper needs large image benchmarks. It is that the paper does not establish a meaningful problem where its additional generality matters, nor the statistical and computational cost of solving that problem. As presented, the contribution is primarily a stochastic-analysis extension whose significance for machine learning remains unclear.

Questions for authors

o. Can the authors demonstrate a conditioning problem that is reachable under the reference dynamics and satisfies their assumptions, but cannot be treated through the usual smooth conditional likelihood or a known intrinsic-coordinate representation? What quantitative advantage does the proposed method provide on that problem?

o. How does the estimation error of the learned control scale with the increment window, dimension, and number of training trajectories? What computational cost is required to achieve a given conditional-sampling accuracy compared with an appropriate baseline?

4 comments

Thank you for your thorough and considerate comments. We have tried to address main concerns raised by you so long as they are feasible and do not break the integrity of our work. We would be happy to discuss/answer any further question or concern that may remain.

Q: Can the authors demonstrate a conditioning problem that is reachable under the reference dynamics and satisfies their assumptions, but cannot be treated through the usual smooth conditional likelihood or a known intrinsic-coordinate representation? What quantitative advantage does the proposed method provide on that problem?

Thanks for asking this great question. Indeed, both examples that we have provided satisfy these conditions. Short answer is that, due to deterministic conditioning at t=Tt=T, ∇xlog⁡p(Y=y∣XT=x)\nabla_x \log p(Y=y|X_T=x) does not exist. So time-reversal is ill-defined. Furthermore, since the initial condition is deterministic, the support of the law of the initial condition is null. As a consequence of the Stroock-Varadhan support theorem, ∇xlog⁡pt(x)\nabla_x \log p_t(x) (the marginal of the law of XX), the score function does not exist at t=0t=0. To learn the score, using Bayes' formula, the usual method is to learn both ∇xlog⁡pt(x∣Y=y)\nabla_x \log p_t(x|Y=y) and ∇log⁡pt(x)\nabla \log p_t(x). However, there is a singularity near t=0t=0; so, one cannot learn ∇log⁡pt(x)\nabla \log p_t(x) via the ``usual'' score network. Both of these singularities near t=0t=0 and t=Tt=T cannot be overcome via local coordinates. The only way out is to mollify these points, which is an intractable problem to do at desired precision. Similar problems also appear in the low-noise regimes, which is actively being tried to avoid, for instance see

Karrast et al. Elucidating the Design Space of Diffusion-Based Generative Models.

Thus, these score-based models technically cannot condition these models perfectly, there always will be a fraction of error (the images will be "noisy"). The second problem is that, score functions do not work when the noise is ``low'', because variance of the score function increases. Our methods work in both cases. In Appendix H.2H.2 we have compared our method with DPM, which does not rely on this for comparison in low noise and demonstrated that in degeneracy and low noise schemes they also work way slower than our method. We have also demonstrated how badly these Bayes' formula based score methods fail in such degeneracy. All these problems were discussed in the prior literature we have cited, mainly in Pidstrigach et al. (2025). Furthermore, especially the singularity of flow/score matching near t=0t=0 is also generally \emph{very well known and well documented}, and thus we did not explicitly mention them. For instance, see Section 33 of

Chung. et al. Diffusion Posterior Sampling For General Noisy Invser Problems.

Q:Scaling and efficient of the estimator

This is more of a neural network question and is specific to the architecture (as well as to the properties of the underlying control, which inform the choice of activation function, etc.). We have provided some elementary results in Appendix G.4 using linear regression (and assuming that it is actually feasible). Answering this question is almost identical to providing a quantitative proxy for the discrepancy between the distribution learned by a generative model and the target distribution.

The central estimator follows a familiar martingale-conditioning argument.

The estimator that we use is the conditional expectation of those increments, which are different from the ``raw" versions. Theoretically, when the diffusion is a strong solution, one can relax Assumption 22 to Brownian filtration (enhanced under X0X_0), which results in an open-loop control instead.

Under the Brownian filtration, one has to take conditional expectation with respect to B[0,t]B_{[0,t]}, which requires one to sample the entire path at any given tt and the resulting control is open-loop. In particular, the martingale representations can actually differ in both cases; these are not the same in the degenerate case, see Theorem 44. Existence of an open-loop control at the same time does not violate our assumptions either, because one considers a process at a larger filtration (Radon Nikodym density ZZ can be taken anticipative). Theorem 4 in particular shows that even when smooth likelihood exists, it induces a closed-loop control. This is because the representation of the exponential tilt induced by ZZ relies on a stochastic integral of the control uu and it is not obvious a priori that this term is adapted even though ZZ is adapted. In the presence of hard constraints, one needs all the terms that appear in construction adapted to the filtration F(X)\mathcal F(X) (for instance in robotics and finance). Sometimes the generated trajectories by the models are as important as the success of the conditioning.

Why degeneracy matters and is natural. What this paper has to do with generative modelling.

In a generative model, we are given an unknown data distribution μdata\mu_{data}, as in any image generation setting. We observe sample vectors X~i\tilde X_i, drawn i.i.d. from this distribution and construct an SDE that uses these samples to generate synthetic data. This is the \emph{purpose of generative modelling}, with applications in fields such as finance, where the diffusion processes that one has to consider can be very singular; see, for instance, problems related to rough path theory.

In practice, however, we do not have access to $ {\mu}_{data}$, but other bigger dimensional data that contains it (for instance when data=images, but we only get pixels). The idea is that the models that learn the images should be lower dimensional, but, we usually get pixel space, which is higher dimensional. In particular, in this sense, it is natural that these diffusion models should be degenerate, otherwise we will memorize a larger space than the image manifold.

These distinctions matter when considering the score of the law of the diffusion. If the initial law is supported on a lower-dimensional set, it need not admit a density with respect to ambient Lebesgue measure, and its ambient score need not be defined. Even when the diffusion admits a smooth density for t>0t>0, its score can become singular as t→0. The discrepancy between the dimension of the ambient space and that of the initial support is one source of this behavior, although the precise singularity also depends on the geometry of the support and the diffusion dynamics. A common heuristic is that this mismatch is the cause of the ``semantic mismatch'' that we encounter in generated images (you learn how to blend pixels together not how images should look like).

In the literature, a common way to avoid this endpoint singularity is to ``smoothen the data'' with highly peaked Gaussians, or to stop the diffusion before reaching the singular endpoint. At a fixed, nonzero smoothing scale, however, one learns a noisy version of the data distribution, rather than the data distribution itself. When we generate samples via time reversal, μdata\mu_{\mathrm{data}} becomes the target distribution, and the singularity associated with the data endpoint remains; its time label simply depends on the convention used for the reversed process. Similarly, bridge models can have singularities at t=0t=0 and t=Tt=T, even when their laws are regular for 0<t<T0<t<T.

The question is then how we can extract \emph{particular} information when these degeneracies are present, especially when that information concerns \emph{rare events}. We believe that this is a very natural problem for generative models. The singularities that we deal with arise naturally in generative modelling, even though we do not provide a new construction of a generative model, per se. To avoid underfitting, it is natural to consider sufficiently expressive models in high-dimensional ambient spaces. When the data remain concentrated on lower-dimensional sets, the endpoint singularities described above become relevant, especially if one is interested in dealing with closed-loop controls.

When the noise is also low, these method still fail in most data, as we mentioned above and demonstrated via newly added examples. So in general, to learn the actual ``data'' more efficiently, we have to overcome these singularities, and one actually has to work with degenerate models in practice to overcome these. We also remark that generative modelling is not a subtopic of image generation, and there was no indication on the conference submissions that this category is restricted to image generation.

These properties of diffusions are known since 70's. In the context of generative modelling, it is also studied in

Jakiw Pidstrigach. Score-Based Generative Models Detect Manifolds. NeurIPS 2022.

Relation between rank deficiency and conditioned likelihood

As a direct consequence of Theorem 4, if drift propagates a noise into those directions (which can happen under Hormander's condition, which requires higher order regularity on top of the bracket condition), this results in an open-loop control, under which we might produce paths that are not ``physically possible'' within the dynamics to reach to the condition. The third paragraph of our introduction explicitly highlights that we are looking for a closed-loop control. closed-loop controls can be shown to be more "robust" in the sense that, in generative models we only learn approximation of the true process, and the learned control can be shown to converge to that of the target process under Gamma converge, a standard argument. Since the open control depends on external noise, the same does not hold (as the "noise" does not have to converge). Even when a smooth conditioned likelihood exists, our methods show that they induce change of measure that is attainable with the current dynamics in this general setting.

Furthermore, to calculate ∇log⁡p(Y=y∣Xt=x)\nabla \log p(Y=y|X_t=x), ``the usual score-based'' method is to approximate ∇log⁡pt(x∣Y=y)\nabla \log p_t(x \mid Y=y) and ∇xlog⁡pt(x)\nabla_x \log p_t(x). However, here we have an additional issue that ∇xlog⁡pt(x)\nabla_x \log p_t(x) might not exist or singular as we have explained above. Hormander's bracket condition is not sufficient for this in general (you need very high order of smoothness on top of the bracket condition). Even if it exists, there might be stability issues in practical applications in the presence of low noise for instance, as discussed in Pidstrigach et al. We have added an extra experiment in Appendix H.2H.2 to demonstrate that in a low noise regimes our method works faster than score-based approach.

Efficiency of our method.

The suggested statistical properties are straightforward, and can be derived by standard Monte Carlo sampling techniques and Ito formula; hence we believe these addendum would be trivial and not interesting by themselves. However, for the sake of completeness, we have included some elementary results in the Appendix G.4G.4. As a consequence of the control formulation we have proposed in Section 5.15.1, one can also obtain more "stable" but computationally expensive derivation of this control using the "Forward Backward SDE" formulation, for instance, see Domingo et al. as we have discussed above.

Experiments

We have explained more explicitly our intention behind the experiment: to depend purely on extrinsic information rather than local information. These experiments were conducted from that viewpoint to simply demonstrate that they work in practice without having to identify the singularities that can occur. It is impossible to cover all edge cases, and our purpose was a demonstration rather than a claim to offer a solution to everything. We have even mentioned this in the subsection on the fashion experiment. However, we have explained how the provided examples actually support our theory, especially related to the singularities at t=0t=0 and t=Tt=T.

Assumption 33 is not necessary in our work; thus, bb, σ\sigma, and YY can depend on paths of XX. We have mentioned this explicitly in the work. In this case, score-based approaches do not paper because one cannot even take the gradient'' of the likelihood function, and our results entail asymptotic convergence. Alternative approaches exist that depend on pathwise derivatives (through the Dupire derivative), but these methods rely on much stronger'' assumptions to simulate these trajectories. We also note that the methods in our work are not pathwise, and we do not claim that they are.

Particular applications of these methods include simulating the survival'' of a system (when $Y=G(X_{[0,T]})$), world models (when $b$ and $\sigma$ depend on paths of $X$), and the simulation of barrier options in finance (when $Y=G(X_{[0,T]})$). Since $\sigma$ can change rank with positive probability at any given snapshot'' of the system, one needs to be aware of all the possibilities that can occur in the system to analyze it via ``intrinsic representations''; this is not viable, for instance, when the system is very large.

Experiments with non-constant diffusion coefficients can now be found in the Appendix H.3H.3.

Reviewer_ennqOpenReview ↗

Thank you for your response

I thank the authors for their detailed responses to my review. I have updated my score.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.