- Rating
- 1
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 1
- Soundness
- 1
Read review
Paper summary
While standard diffusion models use a fixed Gaussian forward process, recent work has used various non-Gaussian forward processes to aid modeling in specific data regimes. This paper proposes to construct a (posterior) distribution over discrete probability measures , which specify a distribution over a scalar diffusion coefficient , which is consequently used to scale the Gaussian noise added to the data at time . Conditioned on , the perturbation law is still Gaussian, but integrating out gives a non-Gaussian perturbation law. The constructed (generalized) posterior favors measures that are likely under the prior and give rise to small score matching losses when trained with it. Experiments show that the posterior shape depends on both the task and training budget.
Critical strengths
- Interesting research direction.
- Learning diffusion models with non-Gaussian forward processes is an important recent research direction which may improve modeling of data with special statistical structure (heavy-tailed, anisotropic etc.).
Critical weaknesses
- Paper is difficult to follow.
- Notation is not properly introduced in many parts of the paper: For example, the introduction mentions , which is only defined in Section 3.2. Some symbols are completely undefined, like the x-axis label in Figure 1 (which should be ), or the "primary generalized-Bayes temperature" in Appendix F.
- Unclear goal.
- While the authors propose to learn the perturbation law, it is never clearly stated what purpose this serves. If the goal is to improve generative performance of the trained model, then both the main posterior construction, as well as the experimental evaluation is fundamentally flawed (see below).
- The corrected risk is difficult to compute.
- The advantage of estimating the DSM loss is precisely that the true score is not needed. However, the proposed corrected risk (which is used to update the generalized posterior over ) loses this property as it needs access to the true score (or, equivalently, the MMSE denoiser ) to evaluate . In other words, estimating is as hard as generative modeling (i.e., obtaining ).
- A measure that minimizes may not be good for generative modeling.
- There is a fundamental issue with the idea of constructing a posterior which favors measures that minimize the expected score risk . The main issue is that there exists which makes the denoising problem "easier", while not learning the data distribution at all: Consider with , which ensures . As , this measure places arbitrarily large amounts of probability mass on . In this case, denoising is trivial, since no noise is added to the data. Under mild regularity conditions, one can thus show that as , a degenerate score network achieves risk, i.e. .
- In fact, this is in line with Theorem 3.5: For a nontrivial data distribution and , one can show that as , which also makes the reverse-path KL diverge. Hence, is favored by the posterior (assuming a wide enough prior), while generative modeling completely fails, and no scores are ever learned by the model.
- Conceptually, it seems more natural to construct a generalized posterior using (i.e., the reverse-path KL), not just .
- Lack of convincing experimental evaluation.
- All experimental results measure corrected score risk, which---as detailed above---is not a good measure for generative performance. Using , we would achieve a corrected score risk of , yet are left with an unusable generative model.
- The authors do not specify the DP-base distribution in their synthetic experiments (Section 4.1), but mention that the "raw atoms lie in " (L1226). Their "non-Gaussian optimal" concentrates mass exactly on the boundary of this interval, yielding and after normalization. This raises the concern that BNP is heavily dependent on the choice of prior. Moreover, it seems like the prior support was chosen such that the degenerate solution discussed above is excluded. It would be beneficial to add ablation studies regarding robustness w.r.t. different prior distributions.
- The comparison between BNP and the baselines (Table 2) is unfair in terms of compute: BNP uses at least the compute to train the models, as described in Section 4.2 (both a score model and a critic are learned to produce the posterior, and finally, a fresh score model is trained from scratch).
Questions for authors
- In Section 4.2, it is unclear to me what measure is used to train the final score network. It is just a sample from the generalized posterior, which is frozen afterwards? I could not find a proper explanation of this procedure.
11 comments
To W3 — Corrected DSM and cross-law comparison:
The obstacle is that the usual DSM objective has the correct score minimizer for each fixed , but its numerical value can misrank different laws. Specifically (Section 2.3),
The irreducible variance is constant with respect to for fixed , yet varies across . Consequently, a lower raw DSM value can reflect an easier-to-predict conditional target rather than a more accurate learned marginal score. Subtracting is therefore required by the law-selection objective itself; it is not an additional metric introduced after observing the results.
The correction has a concrete denoising-regression representation. Writing , it can be estimated using a fitted denoiser. We agree that this raises the crucial question of nuisance-estimation error. Appendix B addresses it directly. Let the held-out corrected loss be the held-out DSM loss minus the estimated correction. For the law-specific cross-fitted construction, Appendix B.2, Eq. (7), shows that, conditional on the complementary training folds, the expected held-out corrected loss equals
,
where is the weighted nuisance excess risk. The residual bias has a specific sign and is law-dependent. Thus, the relevant issue is whether the risk gap between two candidate laws exceeds the difference of their nuisance biases and validation fluctuations. Lemma B.1 decomposes these errors, and Theorem 3.2 gives conditions under which the cross-fitted criterion targets the training-algorithm risk. This is the statistical rationale for correcting DSM while explicitly accounting for estimation error. It also closes the loop with BNP inference: the generalized posterior weights candidate laws by their corrected empirical risks. Theorem 3.3 shows that, when the uniform estimation error, risk gap, and near-optimal prior-mass conditions hold, the ideal generalized posterior concentrates around laws minimizing the learner-dependent finite-resource risk.
We also test this mechanism separately from BNP adaptation. In Figure 1, the exact finite-sample score risk of a controlled fixed-feature learner is available independently of the validation criterion. Among the 36 pairs formed by nine candidate laws, raw DSM reverses 29/36 exact-risk orderings, while the analytically corrected criterion reverses 0/36. The correction in this diagnostic uses the available Bayes denoiser; hence the comparison isolates the law-dependent DSM offset. In an additional one-dimensional nuisance experiment with 16,384 training samples and 20 paired seeds, replacing the initial ridge denoiser with a small MLP trained for up to 1600 epochs reduced ranking reversals exceeding two evaluation standard errors from 21/36 to 0/36, while the maximum cross-law nuisance-excess gap decreased from 0.742 to 0.038. The remaining single sign reversal was within evaluation uncertainty. For CIFAR-10 and HRRR, we instead use a shared ensemble of two fitted nuisance denoisers on held-out evaluation data (Appendix G). The distinction between that practical estimator and the law-specific cross-fitted construction is now explicit in Sections 2.3–2.4.
To W4 — Near-zero-scale example and reverse-process guarantee:
The near-zero-scale example usefully identifies a problem with unrestricted optimization over perturbation laws: making the forward corruption vanish can trivialize a denoising objective. We address it by specifying the domain of comparison, rather than treating mean-one normalization as sufficient on its own. In Section 2.2, consists of mean-one probability laws on a compact interval bounded away from zero. In the primary experiments, raw atoms in yield normalized atoms in . Thus, the proposed collapse through a sequence of vanishing scales is outside this fixed admissible family. The raw-normalized prior has a more restricted image than the entire theoretical class, as Section 2.2 explains.
These two aspects of the theory serve complementary purposes: the admissible class specifies which laws can be compared, while the reverse-process analysis explains why the risk used for that comparison is meaningful for diffusion. For a fixed , our forward process remains an ordinary Brownian diffusion, and the reverse process is formulated conditionally on ; after marginalizing over scales, we use the joint latent-scale/path law. Appendix E now specifies the conditions needed to identify the reverse SDE with the actual time-reversed forward process, construct measurable and unique-in-law learned path kernels, and apply Girsanov with the correct change-of-measure direction. Under Assumption E.1, Theorem 3.5 establishes for :
Here accounts for initializing the learned reverse process from a reference terminal law; Lemma E.2 bounds this mismatch under the stated lower-scale and moment conditions. Data processing then yields an endpoint-distribution bound. The point of this result is precise: the score error used to compare laws is one component of a conditional reverse-process discrepancy, alongside terminal mismatch. The full-data endpoint requires the additional regularity stated in Appendix E.
Thank you for the careful review. Your comments point to an important distinction that we have made more explicit in the revised manuscript: our goal is to learn a perturbation law suited to a given score learner and finite training budget, rather than to identify a universally best noise distribution. We appreciate the opportunity to clarify why this is a meaningful statistical problem, how the corrected criterion makes law selection possible, and what our theory and experiments establish.
To W1–W2 — Statistical objective and perturbation-law learning:
The choice of a perturbation law affects more than the marginal shape of the injected noise. Under our trajectory-level model, a single Λ∼G is held fixed along each path, and the score learner is conditioned on λ. Changing G changes which conditional score fields are encountered and how often the learner must fit each of them. With finite data, model capacity, and optimization steps, this changes the difficulty of the learning problem, even when the average scale is fixed. This motivates our finite-resource target (Section 3.2):
Gm,As⋆∈argminG∈GRm,As(G)
where Rm,As(G) is the expected score risk of the estimator returned by a specified training procedure using m samples. Theorem 3.4 makes the mechanism explicit in a fixed-feature setting: G enters the approximation error, the DSM-target variance, and the leverage-dependent estimation term. Thus the preferred law can depend on the learner and its available resources, not merely on the marginal noise shape.
This is also where BNP inference enters. Because the law is a distribution over trajectory-level scales, a normalized Dirichlet-process prior allows the selection criterion to adapt that distribution rather than restrict the comparison to a prespecified list of Gaussian or Student-t candidates. The mean-one constraint EG[Λ]=1 fixes the average scale while allowing its allocation across noise levels to vary. Crucially, flexibility alone is not the contribution: the corrected risk in Comment 2 supplies a statistically meaningful criterion for comparing such laws. Section 2.2 specifies the compact admissible class on which this comparison is made.