acceptodds
Under review as a conference paper at ICLR 2027

When can diffusion models solve inverse problems accurately?

Abstract

Diffusion models approximate the score of a target distribution to generate new samples. Here we ask how well a diffusion model learns the true data distribution's underlying features by posing this question as an inverse problem. When the target belongs to a parameterized family, can generated samples accurately identify the unknown parameter? We propose sensitivity of the diffusion score matching (DSM) loss as a mechanism for accurate parameter recovery. Specifically, we prove an information-theoretic sufficient condition under which generated samples recover a parameter accurately, even when the target score itself appears insensitive to parameter changes. For example, for Gaussian mixtures in arbitrary dimension, we show that the mixture-weight parameter is recoverable at the same order of accuracy as the DSM loss. This resolves an apparent paradox in the theoretical literature. Although the scores of well-separated Gaussian mixtures can yield poor mixture-weight estimates, we show that scores along the noising process remain sensitive to mixture weights and thereby enable accurate recovery. Experiments on benchmark datasets corroborate this mechanism: noise schedules designed to reduce sensitivity also degrade parameter recovery. Our results characterize when diffusion models can capture rare modes in multimodal mixtures and, more broadly, when diffusion-generated samples can support accurate solutions to scientific inverse problems.

Primary area:
generative models
Keywords:
diffusion models, AI for science

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_LMf5OpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This work studies the estimation of the weights of Gaussian mixtures using diffusion models. Clean-score and intermediate-noise score oracles are compared based on their minimax rates. The authors propose the DSSI sensitivity index, which relates the denoising score-matching loss to squared mixture-weight error for a parametrised distribution family, with a lower bound for two-component Gaussian mixtures. The authors examine score sensitivity across noise levels in Gaussian mixture and MNIST experiments.

Critical strengths

The paper studies an important question of the accuracy of estimation of the parameters of a target distribution based on diffusion-generated samples. The authors motivate this question well via surrogate generative models for downstream inference, where the preservation of mixture weights and component variances matters. The Gaussian mixture example shows how sensitivity to mixture weights changes across different noise levels, motivating the usage of intermediate scores.

Critical weaknesses

The work contains several critical issues in its core components that undermine its main theoretical guarantees and the quantitative interpretation of its experiments.

First, Theorem 1 does not establish the claimed minimax advantage of observing scores at an intermediate diffusion time over observing them at time zero. The oracle score depends on the white noise defined over L2(pt(γ))L^2(p_t^{(\gamma)}), making its covariance also depend on the unknown parameter (Definition 1), but the KL calculation assumes a common covariance (see lines 185–187). This impacts the statistical experiment underlying the theorem. Even if this is fixed, the upper-bound proof has two other problems. The predictive density from the Yang–Barron argument generally lies outside the parametrised family of score-observation laws; hence, the parameter-risk inequality does not apply directly to it. Secondly, the critical-radius equation also contains the noise variance, although the KL metric already includes it. I believe that in a regular Gaussian shift experiment, this normalisation yields an O(σt4log⁡σt−1)O(\sigma_t^4\log \sigma_t^{-1}) risk bound, in contrast to an Ω(σt2)\Omega(\sigma_t^2) lower bound. So the minimax comparison stated by the paper requires a well-defined oracle and, I think, a substantially revised proof.

The theoretical guarantees based on DSSI have problems of a similar magnitude too. Theorem 2 relies on the lower bound in Prop. 3, whose proof discards a possibly nonzero term from an interval endpoint (lines 1338–1349). The proposition is false even in the case when Gaussian components are well separated and β1=20\beta_1=20. Additionally, the proof uses λt=1\lambda_t=1, but the DSM loss uses λt=1−αt\lambda_t=1-\alpha_t, and the calculation necessary for the latter weighting is left unresolved. Simultaneously, the extension to any learned score models conflates the learned score residual with the distribution-level DSM discrepancy between the target and the generated distributions (please see lines 341–347 and 968–990). The denoising identity controls the former, yet a field used in a reverse SDE does not have to equal the scores of the forward marginals of the generated distribution. This transfer may fail even under exact initialisation and exact sampling. Hence, a valid sensitivity inequality between family members would not necessarily establish the stated recovery guarantee for general learned samplers.

I have other concerns, minor in comparison, regarding the experiments, which I try to summarise below. The DSSI estimates for MNIST rely on a finite grid of weights and surrogate scores. With exact scores, the minimum over the grid is an upper bound on DSSI; with surrogate scores, it is not guaranteed to be either an upper or a lower bound. This, in my opinion, is important, as the parameter-error guarantee requires a lower bound. So the claimed fivefold improvement in lines 421–423 is not guaranteed by these estimates. Regarding the schedule ablation, the experiments illustrate sensitivity to noise and schedules, but do not isolate the DSSI contribution from the accompanying change in numerical resolution.

Questions for authors

I believe that addressing the aforementioned theoretical concerns would strengthen the paper, but doing so will result in significant changes to the current submission, requiring another cycle of full review.

10 comments
AuthorsOpenReview ↗

About numerical results

Thank you very much for bringing up the important point about the numerical results. We agree that we cannot compute the true value of DSSI. In addition to the reasons you point out, the definition is also an infimum over an infinite set (of γ\gamma values). So, we can only estimate an upper bound on the infimum numerically. Note that, although we have defined DSSI using analytical scores, this is not necessary for DSSI to be useful as a computational proxy. The neural network approximations of the score at different values of γ\gamma can instead be taken as the ground truth. Hence, our estimates of DSSI are an upper bound on the sensitivity of the neural network scores. In our results we find that as the upper bound increases (when the noise schedule spends less time in the sensitive region), the parameter estimates incur larger errors. This is in qualitative agreement with our theoretical results that sensitivity can be a predictor for parameter recovery.

AuthorsOpenReview ↗

Extending to learned score functions

We completely agree with your objection that the DSSI is a notion that uses the analytical scores of the mixture and hence does not automatically reflect sensitivities in practice. However, consider the scenario where the approximation error and the generalization error are small: that is, the learned score is close to an analytical score in the mixture family. Neglecting estimation error (which can be made small by generating large number of samples with the trained diffusion model), there is some $ gamma'$ in the parameter class corresponding such that ∣∣sNN−sγ′∣∣2=o(∣∣sγ−sNN∣∣2)||s_\mathrm{NN} - s^{\gamma'}||^2 = o(||s^\gamma - s_\mathrm{NN}||^2) for any true parameter value γ.\gamma. In other words, the score error along the mixture family dominates the approximation+learning errors. Then, ∣∣sNN−sγ∣∣2≤(1+ϵ)∣∣sγ−sγ′∣∣2.||s_\mathrm{NN} - s^\gamma||^2 \leq (1+\epsilon) ||s^\gamma - s^{\gamma'}||^2. Thus, we get a larger upper bound on the parameter error: from the DSSI lower bound, we know that, for some C>0,C > 0, ∣γ−γ′∣2≤C∣∣sγ−sγ′∣∣2≤C(1+ϵ)∣∣sγ−sNN∣∣2.|\gamma - \gamma'|^2 \leq C ||s^\gamma - s^{\gamma'}||^2 \leq C (1+\epsilon) ||s^\gamma - s_\mathrm{NN}||^2.

Thus, when the ``projection'' error on to the mixture family (due to approximation and optimization of the score) is small when compared to the overall score error, the DSSI lower bound is informative about parameter error estimated from the generated samples.

I thank the authors for their replies. I would like to reflect briefly on their replies to Reviewer vtB7. I remain unconvinced by the framing in terms of a data-processing paradox. The data-processing inequality states that noising does not increase the information available in the observations, but it does not imply that a particular estimator applied to the original data must outperform a different learning and sampling pipeline using noised data. Thus, better parameter recovery by diffusion than by target-score estimation followed by non-annealed Langevin sampling does not constitute a contradiction. The proposed intermediate-score sensitivity mechanism may explain an advantage within these restricted procedures, but this should be distinguished from an information limitation of the underlying data. I would encourage the authors to frame the comparison accordingly and remove the remaining statements that attribute mixture-weight insensitivity to the original samples.

Now to my points. I agree that fixing a common reference measure is valid, but note that it changes the KL metric and its relationship to the score-matching loss, so the stated identity between KL and ℓSM\ell_{\mathrm{SM}} does not necessarily remain true. I thank the authors for the normalization change, which addresses my σ4\sigma^4 objection. The constant 1111 seems to be obtainable for a finite-net estimator, but the stated risk bound has not been established for the proposed minimization over all γ\gamma. The Fano argument appears to be leading somewhere, but it does not yet establish the claimed minimax comparison, so the claim that “the main result remains the same” needs justification.

The replacement DSSI inequality does not appear to be correct under the stated assumptions. The authors should provide a complete proof under explicitly stated assumptions, including the DSM weighting used in the paper. The transfer to samples generated by the learned model also remains unresolved. The provided argument controls a parameter selected in score space, not the parameter estimated from the generated distribution.

Moreover, reinterpreting neural scores as ground truth defines a different proxy and does not restore the quantitative guarantee. Regarding the schedule versus numerical resolution, the authors have not addressed the confounding identified in my comments.

Given that the major theoretical issues persist, I maintain my decision of “Clear rejection.” Aside from the theoretical aspects, the practical significance of the work is further questioned by Reviewer C8s1, whose concerns I share.

Reviewer_C8s1OpenReview ↗
Rating
2
Confidence
1
Informativeness
1
Presentation
1
Soundness
2
Read review
Paper summary

This paper asks whether samples generated by a diffusion model faithfully preserve the parameters of the data distribution. The authors frame this as recovering the parameter of a parametric family, for example the weight of a mode in a mixture, from generated samples. They argue that the clean score of a well-separated mixture is nearly insensitive to the mixing weight, whereas the scores of the noised marginals at intermediate diffusion times are sensitive to it, and that this is why diffusion models can recover mode proportions when score-based Langevin sampling cannot.

Critical strengths

The paper focuses on the important ground of the diffusion model.

Critical weaknesses

I want to say up front that my background is in image generative models, including score-based diffusion models, and not in learning theory. This review is therefore written from the perspective of a generative-modeling researcher reading the paper from outside its core area. In its current state of writing and presentation, I was unable to follow most of the technical development, especially Section 3, and I cannot guarantee the correctness or significance of Theorems 1 and 2. My comments focus on how accessible the paper is to the broader ICLR generative-modeling audience and on the empirical evidence, and I have set my confidence accordingly.

The title and framing were misleading to me. In the image generative modeling literature, “diffusion models for inverse problems” usually refers to recovering an image from measurements, as in super-resolution, deblurring, inpainting or compressed sensing. I read a large part of the paper expecting such a task before realizing that “inverse problem” here means estimating a parameter of the training distribution from generated samples. I expect many diffusion-model readers at ICLR to have the same experience. Could the authors state this meaning explicitly in the abstract and introduction, and consider a title that reflects the actual question? I believe this paper will be beneficial if it clarifies its problem setup in the early introduction.

The problem setting is never stated plainly, and the context comes too late. It took me a long time to understand what is being estimated, from what, and by whom. My best understanding is that an unconditional diffusion model is trained on samples from p^(\gamma*), new samples are generated, and \gamma is then estimated from those samples. If that is correct, a concrete description of this pipeline early in the introduction, ideally with a diagram, would help. I was also unsure whether a network predicts the parameter. As far as I can tell, none does, and “parameter recovery” is an evaluation of the generated samples. Could the authors move the related work into the introduction and state clearly which part of the question is new?

Method section and theory part is not accessible to non-specialists. With the given introduction and preliminary writing, I could not follow Section 3. The specific places where I got lost were starting from line 165. I also could not tell how Theorem 1 relates to a trained network. Before each formal statement, an intuitive explanation of the assumptions and of the main theorem, illustrated on a worked Gaussian-mixture example, would make the paper accessible to the wider diffusion community. As written, I am unable to assess the correctness or significance of Theorems 1 and 2.

The experiments are limited. The empirical evidence consists of a two-component Gaussian mixture in 10 dimensions and a 12-dimensional autoencoder latent space of MNIST digits 1 and 8. Only one target weight and three noise schedules are tested, and no variability across runs is reported. The abstract's statement that “experiments on benchmark datasets corroborate this mechanism” therefore seems overstated. The introduction motivates the work with scientific applications, but none is evaluated. Could the authors add one scientific example matching the motivation?

Overall, I found this paper pretty hard to read for me. Although my actual evaluation is one score below a weak rejection, I give a weak rejection to ensure that discussion among other reviewers in this research field is weighted more heavily.

Questions for authors

Please refer to the weaknesses section above.

3 comments
AuthorsOpenReview ↗

Comments on inverse problems

Thank you very much for pointing out specific issues and with the readability of our paper! We completely agree that the exact problem needs to be stated clearly, and we have added a line in the introduction: we are interested in studying how accurately a parameter can be recovered from diffusion-generated samples. In a mathematical sense, in an inverse problem, we need to recover an input from observations of an output. Your understanding of the pipeline (sample from the target distribution -> train a diffusion model -> generate new samples -> estimate unknown parameter) is absolutely correct. We will include this pipeline in the text in the introduction.

It is common in scientific applications to consider parameters of the training/data distribution to be an input and here, the outputs here are the diffusion-generated samples. For instance, in AI emulators for the climate/weather, there are many unknown parameters corresponding to geophysical phenomena that is described the data (atmospheric and surface variables). The goal is to understand if by training a diffusion model practically can help us identify these parameters. In another example, a generative model for some biological features extracted from pathology images should help us recover unknown parameters of the underlying (pathology) image distribution.

Our paper is focused on how accurately such recovery is possible, on the theoretical side. The main goal of our paper is to explain when the diffusion-generated samples correspond to a distribution with the unknown parameter being very close to the true parameter. In other words, our goal is to propose a mechanism and explain the underlying theory behind when practitioners can use the estimated parameter (solution of the inverse problem) from diffusion models for downstream applications. The true parameter γ∗\gamma^* describes a property of the training distribution, such as its mixture weight. Parameter recovery therefore is a very practically relevant evaluation of whether the generated distribution preserves properties of the distribution. We will summarize the main results of our two theorems and we will add a similar elaboration in the paper, before the technical developments of the two theorems.

  1. In Theorem 1, we show a sufficient condition under which parameter recovery can achieved provably better with the noisy scores when compared to the target score. This condition is expressed in terms of ``score sensitivity'': how much the score varies as we make changes to the parameter. In our next result, we show how large this sensitivity can be for a bimodal Gaussian mixture.

  2. In Theorem 2, we show that the sensitivity of the score function to parameter changes, when integrated over noise scales, is lower bounded by a constant for bimodal Gaussian mixtures. This means that we can upper bound the error in the recovered parameter from the generated samples by a constant x the diffusion score error.

Our experiments provide controlled illustrations of our proposed mechanism (score sensitivity) affecting parameter recovery. In the MNIST experiments, we consider various noise schedules designed to reduce the time the denoising spends in the sensitive region. We find that the recovered parameter has a larger error the lesser the time spent in the sensitive region. This observation corroborates that the sensitivity of the score across the noise scales of the sampling scheme affects how well the generated distribution captures the unknown parameter. The scope of our paper does not include performing large scale scientific experiments to illustrate our theory, but we do believe our results provide sufficient evidence to suggest that the computable quantity, DSSI, is indeed predictive of the accuracy of the recovered parameter.

We would nevertheless like to clarify the scope of the existing evidence. Figures 2(b) and 3 examine sensitivity at several target mixture weights; Figure 2(c) examines modifications of both linear and cosine schedules; and Appendix E.4 examines sensitivity to parameters controlling component means and covariances. We will make these results easier to locate while distinguishing sensitivity analyses from generated-sample recovery experiments. We defer to scientific applications (e.g., with GenCast, a diffusion model for the weather) in future work.

Thank you to the authors for their responses. As I cannot see the revised version of the paper at this stage, I cannot provide further comments on the paper's readability. Also, given the amount of revision needed and that I cannot check the revision now, I will keep my presentation and informativeness evaluation as is.

I also keep my evaluation at 2: Weak rejection, with the explicit note that I am from outside this field.

Thank you for your response -- we appreciate your time and effort despite your different area of expertise.

Reviewer_vtB7OpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

Diffusion models are trained to approximate the score function of a target distribution over various noise levels. This paper investigates whether such models can accurately recover parameters of the target distribution, focusing on the relative weights of two modes as a case study. The authors observe that, when modes are well separated, the score at the target distribution is largely insensitive to the mode weights, while scores at intermediate noise levels remain sensitive. They argue this explains why diffusion models can recover mode weights correctly. The paper introduces a sensitivity index as a diagnostic tool, establishes a lower bound for this index for two-component Gaussian mixtures, and provides experimental evidence that noise schedules that narrow the sensitive noise range impair mode-weight recovery, even though visual sample quality remains unaffected.

Critical strengths
  • Evaluating parameter recovery is a meaningful criterion, especially for scientific applications of generative models.
  • The finding that accelerated sampling preserves mode geometry but not mode mass is insightful and is clearly demonstrated in the paper.
  • The proposed sensitivity index is both straightforward and easy to compute.
Critical weaknesses
  1. The insensitivity to mode weight is actually a property of the score-matching estimator, not of the data itself. The samples from a separated mixture readily reveal the weights. However, the paper frames this as an inherent data issue.
  2. Theorem 1 considers the score plus white noise with an exogenous level σt\sigma_t; the separation between time 00 and time tt seems built into the assumptions rather than derived.
  3. The sensitivity index employs uniform weighting over tt, making it sensitive to time reparameterization, even though the underlying continuous dynamics are unaffected. As a result, the schedule experiments at fixed T = 100 may essentially be measuring the number of steps within the sensitive region.
Questions for authors
  1. How should we choose σt\sigma_t in the oracle model, and what does it represent for a network trained on typical data samples?
  2. Does the effect of the noise schedule observed in Figures 2c and 4 diminish as the number of sampling steps increases?
5 comments
AuthorsOpenReview ↗

Data issue vs score-matching estimator issue

Thank you very much for an astute reading of our paper! Your understanding of the paper is absolutely correct and your questions have led to substantial improvements to the paper. They have also given us an opportunity to clarify some aspects of our results, which may not have come across clearly. In this comment, we respond to the first ``critical weakness'' you have pointed out.

You are completely correct that the data from well-separated mixtures is still sensitive to perturbations to the weights, and it is the score matching estimator that is insensitive to the weight parameter. The existing literature also points to the same failure of the score matching estimator. The problem though, is that, this does not explain why diffusion models are able to recover the parameter accurately when they are able to do so. In other words, why are diffusion-generated samples are better at estimating the parameter when compared to the Langevin-generated samples? At first glance, both seem to be using the score matching estimator. That is, we use the data to estimate the target score and the target score is then used to generate samples, in both cases. Why then, if the score-matching estimator is not good at distinguishing parameters, can diffusion-generated samples still recover the parameter accurately?

Now, the difference between Langevin generated samples and diffusion-generated samples is that the later uses the scores of the annealed distributions. Thus we may be led to believe that adding noise is what helps diffusion models recover parameters. But then, from the data processing inequality: KL(ptγ∣∣ptγ′)≤KL(pγ∣∣pγ′).\mathrm{KL}(p_t^\gamma||p_t^{\gamma'}) \leq \mathrm{KL}(p^\gamma||p^{\gamma'}). That is, as the mode separation decreases (as we add noise), the data can distinguish γ\gamma and γ′\gamma' less and less. This leads us to a contradiction: the noised samples do not help as much as the original samples from the well-separated modes and yet, we find that diffusion models are better at parameter recovery compared to learning the score of pγp^\gamma from the data and then performing Langevin sampling and then estimating the parameter from Langevin sampling. This is exactly the conundrum that our paper resolves.

Our resolution is that the inverse problem of parameter recovery from samples using diffusion models can be thought of in the following way: we first learn not just the target score, but the scores of a sequence of noised distributions, from the generated samples. (We will discuss the effect of the learning error, σt\sigma_t, that you have asked about, in a different comment). Then, from this sequence of scores, we consider estimators for the parameter (e.g., denoising + EM algorithm). The point is that, because the intermediate scores may be more sensitive to the parameter than the target score, an estimator for the parameter that uses the intermediate instead of the target score alone can perform better. This explanation is what we also verify in our numerical experiments.

AuthorsOpenReview ↗

About the score noise model

The score model is designed to describe a diffusion model as a ``score oracle''. The scores available to a diffusion model are not the actual scores, stγ,s^\gamma_t, at a noise level t,t, but rather an approximate score. This approximate score is what we write as stγ+σtW.s^\gamma_t + \sigma_t W. We can therefore think of σt\sigma_t as representing a learning error at noise level t.t.

You are correct -- we have assumed that the separation between parameters as measured by the score distribution, qtγq^\gamma_t is higher at scale tt than at 0. That is, we have assumed that KL(qtγ∣∣qtγ′)≥KL(qγ∣∣qγ′)KL(q^\gamma_t || q_t^{\gamma'}) \geq KL(q^\gamma || q^{\gamma'}) However, this essentially means, because of our score model definition that the sensitivity of score at time tt is greater than the score sensitivity at time 0 (target score sensitivity), as we show: ℓSM(ptγ′,ptγ)≥ℓSM(pγ′,pγ).\ell_\mathrm{SM}(p_t^{\gamma'}, p_t^\gamma) \geq \ell_\mathrm{SM}(p^{\gamma'}, p^\gamma). This is the condition for what we want to prove: that greater intermediate time score sensitivity implies more accurate parameter recovery.

In general, it is difficult to characterize the class of distributions where larger intermediate score sensitivity may occur. We are able to analytically derive an expression for the score sensitivity in the bimodal Gaussian case, and hence a bimodal Gaussian mixture family is an example. Intuitively, here, as we add noise to the distribution, the two modes approach each other and when the modes are close to overlapping is when the sensitivity is maximized. Because it is difficult to derive this condition for a general family of mixtures, we have proposed to check the ``computable'' version of this condition: DSSI. We hope this answers your question, please let us know if this is not what you meant and we will try again!

AuthorsOpenReview ↗

Effect of noise schedule

Thank you very much for your question about our sensitivity index definition. You are absolutely correct that the sensitivity index measures ultimately the number of timesteps/noise scales rather than the ''fraction of time'' spent in the sensitive region. We will say this explicitly in the revision. In the numerical experiments, we write ''fraction of time'' spent but as you point out, this should be the number of timesteps or the number of noise scales in the sensitive region that affects parameter recovery.

We must also clarify that our theoretical results are valid for any noising scheme. In Theorem 1, we compare recoverability of the parameter from the score at time or noise scale in the sensitive region to that from time 0 (clean score). Since this is just comparing two noise scales and not integrating over noise scales, this result is unaffected. Our Theorem 2 (lower bound on the sensitivity index for bimodal Gaussians) can be extended to any noising scheme -- this is because the noising scheme only affects a scalar factor in the integrand that we analytically lower bound.

Finally, we will now answer your question on the effect of noise schedule. First, let us assume that for any DDPM timestepping scheme, as long as the minimum timestep is sufficiently small, the dynamics of the reverse process approximates the underlying SDE. Thus, whenever there is score sensitivity at some t, the process does spend time in that region. However, for a uniform timestepping scheme, the fraction of time spent in the different noise schedules depends on the function t→αtt \to \alpha_t. For instance, a linear schedule spends only 10% of the time in the low noise regime compared to 20% spent by a cosine schedule. Thus, for a uniform time discretization, we still see a change in L(γ∗)L(\gamma^*) and therefore in the figures 2c and 4, even as the number of time steps becomes large.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Request to read and respond to the rebuttal

Dear Reviewers,

We have addressed all your questions and concerns in the responses below. Could you please read them and let us know if we any of your questions remain unanswered or if our answers are unclear? Thank you very much for your reviews and engagement!

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.