- Rating
- 1
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
This work studies the estimation of the weights of Gaussian mixtures using diffusion models. Clean-score and intermediate-noise score oracles are compared based on their minimax rates. The authors propose the DSSI sensitivity index, which relates the denoising score-matching loss to squared mixture-weight error for a parametrised distribution family, with a lower bound for two-component Gaussian mixtures. The authors examine score sensitivity across noise levels in Gaussian mixture and MNIST experiments.
Critical strengths
The paper studies an important question of the accuracy of estimation of the parameters of a target distribution based on diffusion-generated samples. The authors motivate this question well via surrogate generative models for downstream inference, where the preservation of mixture weights and component variances matters. The Gaussian mixture example shows how sensitivity to mixture weights changes across different noise levels, motivating the usage of intermediate scores.
Critical weaknesses
The work contains several critical issues in its core components that undermine its main theoretical guarantees and the quantitative interpretation of its experiments.
First, Theorem 1 does not establish the claimed minimax advantage of observing scores at an intermediate diffusion time over observing them at time zero. The oracle score depends on the white noise defined over , making its covariance also depend on the unknown parameter (Definition 1), but the KL calculation assumes a common covariance (see lines 185–187). This impacts the statistical experiment underlying the theorem. Even if this is fixed, the upper-bound proof has two other problems. The predictive density from the Yang–Barron argument generally lies outside the parametrised family of score-observation laws; hence, the parameter-risk inequality does not apply directly to it. Secondly, the critical-radius equation also contains the noise variance, although the KL metric already includes it. I believe that in a regular Gaussian shift experiment, this normalisation yields an risk bound, in contrast to an lower bound. So the minimax comparison stated by the paper requires a well-defined oracle and, I think, a substantially revised proof.
The theoretical guarantees based on DSSI have problems of a similar magnitude too. Theorem 2 relies on the lower bound in Prop. 3, whose proof discards a possibly nonzero term from an interval endpoint (lines 1338–1349). The proposition is false even in the case when Gaussian components are well separated and . Additionally, the proof uses , but the DSM loss uses , and the calculation necessary for the latter weighting is left unresolved. Simultaneously, the extension to any learned score models conflates the learned score residual with the distribution-level DSM discrepancy between the target and the generated distributions (please see lines 341–347 and 968–990). The denoising identity controls the former, yet a field used in a reverse SDE does not have to equal the scores of the forward marginals of the generated distribution. This transfer may fail even under exact initialisation and exact sampling. Hence, a valid sensitivity inequality between family members would not necessarily establish the stated recovery guarantee for general learned samplers.
I have other concerns, minor in comparison, regarding the experiments, which I try to summarise below. The DSSI estimates for MNIST rely on a finite grid of weights and surrogate scores. With exact scores, the minimum over the grid is an upper bound on DSSI; with surrogate scores, it is not guaranteed to be either an upper or a lower bound. This, in my opinion, is important, as the parameter-error guarantee requires a lower bound. So the claimed fivefold improvement in lines 421–423 is not guaranteed by these estimates. Regarding the schedule ablation, the experiments illustrate sensitivity to noise and schedules, but do not isolate the DSSI contribution from the accompanying change in numerical resolution.
Questions for authors
I believe that addressing the aforementioned theoretical concerns would strengthen the paper, but doing so will result in significant changes to the current submission, requiring another cycle of full review.
10 comments
Extending to learned score functions
We completely agree with your objection that the DSSI is a notion that uses the analytical scores of the mixture and hence does not automatically reflect sensitivities in practice. However, consider the scenario where the approximation error and the generalization error are small: that is, the learned score is close to an analytical score in the mixture family. Neglecting estimation error (which can be made small by generating large number of samples with the trained diffusion model), there is some $ gamma'$ in the parameter class corresponding such that for any true parameter value In other words, the score error along the mixture family dominates the approximation+learning errors. Then, Thus, we get a larger upper bound on the parameter error: from the DSSI lower bound, we know that, for some
Thus, when the ``projection'' error on to the mixture family (due to approximation and optimization of the score) is small when compared to the overall score error, the DSSI lower bound is informative about parameter error estimated from the generated samples.
I thank the authors for their replies. I would like to reflect briefly on their replies to Reviewer vtB7. I remain unconvinced by the framing in terms of a data-processing paradox. The data-processing inequality states that noising does not increase the information available in the observations, but it does not imply that a particular estimator applied to the original data must outperform a different learning and sampling pipeline using noised data. Thus, better parameter recovery by diffusion than by target-score estimation followed by non-annealed Langevin sampling does not constitute a contradiction. The proposed intermediate-score sensitivity mechanism may explain an advantage within these restricted procedures, but this should be distinguished from an information limitation of the underlying data. I would encourage the authors to frame the comparison accordingly and remove the remaining statements that attribute mixture-weight insensitivity to the original samples.
Now to my points. I agree that fixing a common reference measure is valid, but note that it changes the KL metric and its relationship to the score-matching loss, so the stated identity between KL and does not necessarily remain true. I thank the authors for the normalization change, which addresses my objection. The constant seems to be obtainable for a finite-net estimator, but the stated risk bound has not been established for the proposed minimization over all . The Fano argument appears to be leading somewhere, but it does not yet establish the claimed minimax comparison, so the claim that “the main result remains the same” needs justification.
The replacement DSSI inequality does not appear to be correct under the stated assumptions. The authors should provide a complete proof under explicitly stated assumptions, including the DSM weighting used in the paper. The transfer to samples generated by the learned model also remains unresolved. The provided argument controls a parameter selected in score space, not the parameter estimated from the generated distribution.
Moreover, reinterpreting neural scores as ground truth defines a different proxy and does not restore the quantitative guarantee. Regarding the schedule versus numerical resolution, the authors have not addressed the confounding identified in my comments.
Given that the major theoretical issues persist, I maintain my decision of “Clear rejection.” Aside from the theoretical aspects, the practical significance of the work is further questioned by Reviewer C8s1, whose concerns I share.
About numerical results
Thank you very much for bringing up the important point about the numerical results. We agree that we cannot compute the true value of DSSI. In addition to the reasons you point out, the definition is also an infimum over an infinite set (of γ values). So, we can only estimate an upper bound on the infimum numerically. Note that, although we have defined DSSI using analytical scores, this is not necessary for DSSI to be useful as a computational proxy. The neural network approximations of the score at different values of γ can instead be taken as the ground truth. Hence, our estimates of DSSI are an upper bound on the sensitivity of the neural network scores. In our results we find that as the upper bound increases (when the noise schedule spends less time in the sensitive region), the parameter estimates incur larger errors. This is in qualitative agreement with our theoretical results that sensitivity can be a predictor for parameter recovery.