acceptodds
Under review as a conference paper at ICLR 2027

Bayesian Nonparametric Diffusion Models

Abstract

Score-based diffusion models have achieved strong generative performance across a wide range of applications, but their perturbation law, which determines the noisy states and score targets, is typically fixed to a standard Gaussian form. Recent work shows that alternative perturbation laws can improve modeling in specific data regimes. However, existing approaches either prescribe a perturbation family or learn restricted forward-process parameterizations, leaving open how to adapt the perturbation law itself to a particular learner and training regime. We address this problem with a Bayesian nonparametric (BNP) diffusion model that learns the perturbation distribution directly. We show that standard denoising score matching (DSM) contains a perturbation-dependent irreducible term and derive a corrected criterion for generalized Bayesian learning of the perturbation distribution. Theoretically, we show that the corrected criterion consistently targets finite-resource score risk, that the resulting generalized posterior concentrates around the corresponding optimal perturbation laws, and that the same score risk controls reverse-process error. Empirically, controlled experiments show that BNP adapts across Gaussian and non-Gaussian regimes and that the correction enables more accurate comparisons of score error across perturbation laws. On CIFAR-10 and HRRR, comparisons based on estimated corrected score risk show that the preferred perturbation varies across tasks and training budgets. Together, these findings support treating the perturbation law as a learnable component of diffusion models, adapted to the score learner and available training resources.

Primary area:
generative models
Keywords:
diffusion models, score-based generative models, denoising score matching, perturbation learning, Bayesian nonparametrics, generalized Bayesian inference, finite-resource learning

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_9REXOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

While standard diffusion models use a fixed Gaussian forward process, recent work has used various non-Gaussian forward processes to aid modeling in specific data regimes. This paper proposes to construct a (posterior) distribution over discrete probability measures GG, which specify a distribution over a scalar diffusion coefficient λ\lambda, which is consequently used to scale the Gaussian noise added to the data at time tt. Conditioned on λ\lambda, the perturbation law is still Gaussian, but integrating out λ\lambda gives a non-Gaussian perturbation law. The constructed (generalized) posterior favors measures GG that are likely under the prior and give rise to small score matching losses when trained with it. Experiments show that the posterior shape depends on both the task and training budget.

Critical strengths
  • Interesting research direction.
    • Learning diffusion models with non-Gaussian forward processes is an important recent research direction which may improve modeling of data with special statistical structure (heavy-tailed, anisotropic etc.).
Critical weaknesses
  • Paper is difficult to follow.
    • Notation is not properly introduced in many parts of the paper: For example, the introduction mentions Rm,As\mathfrak{R}_{m,\mathcal{A}_s}, which is only defined in Section 3.2. Some symbols are completely undefined, like the x-axis label δ\delta in Figure 1 (which should be hh), or the "primary generalized-Bayes temperature" η\eta in Appendix F.
  • Unclear goal.
    • While the authors propose to learn the perturbation law, it is never clearly stated what purpose this serves. If the goal is to improve generative performance of the trained model, then both the main posterior construction, as well as the experimental evaluation is fundamentally flawed (see below).
  • The corrected risk is difficult to compute.
    • The advantage of estimating the DSM loss is precisely that the true score stλ(Xtλ)s_t^\lambda(X_t^\lambda) is not needed. However, the proposed corrected risk (which is used to update the generalized posterior over GG) loses this property as it needs access to the true score (or, equivalently, the MMSE denoiser rtλr_t^\lambda) to evaluate Cτ(G)\mathcal{C}_{\tau}(G). In other words, estimating Cτ(G)\mathcal{C}_\tau(G) is as hard as generative modeling (i.e., obtaining rtλr_t^\lambda).
  • A measure GG that minimizes Rm,As\mathfrak{R}_{m,\mathcal{A}_s} may not be good for generative modeling.
    • There is a fundamental issue with the idea of constructing a posterior which favors measures GG that minimize the expected score risk Rm,As\mathfrak{R}_{m,\mathcal{A}_s}. The main issue is that there exists GG which makes the denoising problem "easier", while not learning the data distribution at all: Consider Gϵ=(1−ϵ)δϵ+ϵδλlargeG_\epsilon = (1-\epsilon)\delta_\epsilon+\epsilon\delta_{\lambda_{\mathrm{large}}} with λlarge=1−(1−ϵ)ϵϵ\lambda_{\mathrm{large}} = \frac{1-(1-\epsilon)\epsilon}{\epsilon}, which ensures EΛ∼Gϵ[Λ]=1\mathbb E_{\Lambda \sim G_\epsilon}[\Lambda] = 1. As ϵ→0\epsilon \to 0, this measure places arbitrarily large amounts of probability mass on Λ=0\Lambda = 0. In this case, denoising is trivial, since no noise is added to the data. Under mild regularity conditions, one can thus show that as ϵ→0\epsilon \to 0, a degenerate score network sθ≡0s_\theta \equiv 0 achieves 00 risk, i.e. Rscore,τ(Gϵ,sθ)→0\mathcal{R}_{\text{score},\tau}(G_\epsilon, s_{\theta}) \to 0.
    • In fact, this is in line with Theorem 3.5: For a nontrivial data distribution and aT>0a_T > 0, one can show that DT(Gϵ)→∞\mathcal D_T(G_\epsilon) \to \infty as ϵ→0\epsilon \to 0, which also makes the reverse-path KL diverge. Hence, GϵG_\epsilon is favored by the posterior (assuming a wide enough prior), while generative modeling completely fails, and no scores are ever learned by the model.
    • Conceptually, it seems more natural to construct a generalized posterior using DT+Rscore,τ\mathcal{D}_T + \mathcal{R}_{\text{score}, \tau} (i.e., the reverse-path KL), not just Rscore,τ\mathcal{R}_{\text{score}, \tau}.
  • Lack of convincing experimental evaluation.
    • All experimental results measure corrected score risk, which---as detailed above---is not a good measure for generative performance. Using GϵG_\epsilon, we would achieve a corrected score risk of 00, yet are left with an unusable generative model.
    • The authors do not specify the DP-base distribution HH in their synthetic experiments (Section 4.1), but mention that the "raw atoms lie in [0.5,2][0.5, 2]" (L1226). Their "non-Gaussian optimal" G⋆G^\star concentrates mass exactly on the boundary of this interval, yielding δ0.48\delta_{0.48} and δ1.95\delta_{1.95} after normalization. This raises the concern that BNP is heavily dependent on the choice of prior. Moreover, it seems like the prior support was chosen such that the degenerate solution discussed above is excluded. It would be beneficial to add ablation studies regarding robustness w.r.t. different prior distributions.
    • The comparison between BNP and the baselines (Table 2) is unfair in terms of compute: BNP uses at least 3×3\times the compute to train the models, as described in Section 4.2 (both a score model and a critic rr are learned to produce the posterior, and finally, a fresh score model is trained from scratch).
Questions for authors
  • In Section 4.2, it is unclear to me what measure GG is used to train the final score network. It is just a sample from the generalized posterior, which is frozen afterwards? I could not find a proper explanation of this procedure.
11 comments

Thank you for the careful review. Your comments point to an important distinction that we have made more explicit in the revised manuscript: our goal is to learn a perturbation law suited to a given score learner and finite training budget, rather than to identify a universally best noise distribution. We appreciate the opportunity to clarify why this is a meaningful statistical problem, how the corrected criterion makes law selection possible, and what our theory and experiments establish.

To W1–W2 — Statistical objective and perturbation-law learning:

The choice of a perturbation law affects more than the marginal shape of the injected noise. Under our trajectory-level model, a single Λ∼G\Lambda\sim G is held fixed along each path, and the score learner is conditioned on λ\lambda. Changing GG changes which conditional score fields are encountered and how often the learner must fit each of them. With finite data, model capacity, and optimization steps, this changes the difficulty of the learning problem, even when the average scale is fixed. This motivates our finite-resource target (Section 3.2):

Gm,As⋆∈arg⁡min⁡G∈GRm,As(G)G^\star_{m,\mathcal A_s}\in\arg\min_{G\in\mathcal G}\mathfrak R_{m,\mathcal A_s}(G)

where Rm,As(G)\mathfrak R_{m,\mathcal A_s}(G) is the expected score risk of the estimator returned by a specified training procedure using mm samples. Theorem 3.4 makes the mechanism explicit in a fixed-feature setting: GG enters the approximation error, the DSM-target variance, and the leverage-dependent estimation term. Thus the preferred law can depend on the learner and its available resources, not merely on the marginal noise shape.

This is also where BNP inference enters. Because the law is a distribution over trajectory-level scales, a normalized Dirichlet-process prior allows the selection criterion to adapt that distribution rather than restrict the comparison to a prespecified list of Gaussian or Student-tt candidates. The mean-one constraint EG[Λ]=1\mathbb E_G[\Lambda]=1 fixes the average scale while allowing its allocation across noise levels to vary. Crucially, flexibility alone is not the contribution: the corrected risk in Comment 2 supplies a statistically meaningful criterion for comparing such laws. Section 2.2 specifies the compact admissible class on which this comparison is made.

To W3 — Corrected DSM and cross-law comparison:

The obstacle is that the usual DSM objective has the correct score minimizer for each fixed GG, but its numerical value can misrank different laws. Specifically (Section 2.3),

LDSM,τ(G,s)=Rscore,τ(G,s)+Cτ(G).\mathcal L_{\mathrm{DSM},\tau}(G,s) =\mathcal R_{\mathrm{score},\tau}(G,s)+C_\tau(G).

The irreducible variance Cτ(G)C_\tau(G) is constant with respect to ss for fixed GG, yet varies across GG. Consequently, a lower raw DSM value can reflect an easier-to-predict conditional target rather than a more accurate learned marginal score. Subtracting Cτ(G)C_\tau(G) is therefore required by the law-selection objective itself; it is not an additional metric introduced after observing the results.

The correction has a concrete denoising-regression representation. Writing rtλ(x)=E[X0∣Xtλ=x]r_t^\lambda(x)=\mathbb E[X_0\mid X_t^\lambda=x], it can be estimated using a fitted denoiser. We agree that this raises the crucial question of nuisance-estimation error. Appendix B addresses it directly. Let the held-out corrected loss be the held-out DSM loss minus the estimated correction. For the law-specific cross-fitted construction, Appendix B.2, Eq. (7), shows that, conditional on the complementary training folds, the expected held-out corrected loss equals

Rscore,τ(G,s^G(−k))−Eden(G,r^G(−k))\mathcal R_{\mathrm{score},\tau}(G,\widehat s_G^{(-k)})-\mathcal E_{\mathrm{den}}(G,\widehat r_G^{(-k)}),

where Eden≥0\mathcal E_{\mathrm{den}}\ge0 is the weighted nuisance excess risk. The residual bias has a specific sign and is law-dependent. Thus, the relevant issue is whether the risk gap between two candidate laws exceeds the difference of their nuisance biases and validation fluctuations. Lemma B.1 decomposes these errors, and Theorem 3.2 gives conditions under which the cross-fitted criterion targets the training-algorithm risk. This is the statistical rationale for correcting DSM while explicitly accounting for estimation error. It also closes the loop with BNP inference: the generalized posterior weights candidate laws by their corrected empirical risks. Theorem 3.3 shows that, when the uniform estimation error, risk gap, and near-optimal prior-mass conditions hold, the ideal generalized posterior concentrates around laws minimizing the learner-dependent finite-resource risk.

We also test this mechanism separately from BNP adaptation. In Figure 1, the exact finite-sample score risk of a controlled fixed-feature learner is available independently of the validation criterion. Among the 36 pairs formed by nine candidate laws, raw DSM reverses 29/36 exact-risk orderings, while the analytically corrected criterion reverses 0/36. The correction in this diagnostic uses the available Bayes denoiser; hence the comparison isolates the law-dependent DSM offset. In an additional one-dimensional nuisance experiment with 16,384 training samples and 20 paired seeds, replacing the initial ridge denoiser with a small MLP trained for up to 1600 epochs reduced ranking reversals exceeding two evaluation standard errors from 21/36 to 0/36, while the maximum cross-law nuisance-excess gap decreased from 0.742 to 0.038. The remaining single sign reversal was within evaluation uncertainty. For CIFAR-10 and HRRR, we instead use a shared ensemble of two fitted nuisance denoisers on held-out evaluation data (Appendix G). The distinction between that practical estimator and the law-specific cross-fitted construction is now explicit in Sections 2.3–2.4.

To W4 — Near-zero-scale example and reverse-process guarantee:

The near-zero-scale example usefully identifies a problem with unrestricted optimization over perturbation laws: making the forward corruption vanish can trivialize a denoising objective. We address it by specifying the domain of comparison, rather than treating mean-one normalization as sufficient on its own. In Section 2.2, G\mathcal G consists of mean-one probability laws on a compact interval bounded away from zero. In the primary experiments, raw atoms in [0.5,2][0.5,2] yield normalized atoms in [0.25,4][0.25,4]. Thus, the proposed collapse through a sequence of vanishing scales is outside this fixed admissible family. The raw-normalized prior has a more restricted image than the entire theoretical class, as Section 2.2 explains.

These two aspects of the theory serve complementary purposes: the admissible class specifies which laws can be compared, while the reverse-process analysis explains why the risk used for that comparison is meaningful for diffusion. For a fixed λ\lambda, our forward process remains an ordinary Brownian diffusion, and the reverse process is formulated conditionally on λ\lambda; after marginalizing over scales, we use the joint latent-scale/path law. Appendix E now specifies the conditions needed to identify the reverse SDE with the actual time-reversed forward process, construct measurable and unique-in-law learned path kernels, and apply Girsanov with the correct change-of-measure direction. Under Assumption E.1, Theorem 3.5 establishes for 0<τ<T0<\tau<T:

KL(PGrev∥QG,s^rev)=DT(G)+Rscore,τ(G,s^).\mathrm{KL}\left(\mathbf P_G^{\mathrm{rev}}\middle\|\mathbf Q_{G,\widehat s}^{\mathrm{rev}}\right) =\mathcal D_T(G)+\mathcal R_{\mathrm{score},\tau}(G,\widehat s).

Here DT(G)\mathcal D_T(G) accounts for initializing the learned reverse process from a reference terminal law; Lemma E.2 bounds this mismatch under the stated lower-scale and moment conditions. Data processing then yields an endpoint-distribution bound. The point of this result is precise: the score error used to compare laws is one component of a conditional reverse-process discrepancy, alongside terminal mismatch. The full-data endpoint τ=0\tau=0 requires the additional regularity stated in Appendix E.

Reviewer_wJkBOpenReview ↗
Rating
3(+1)
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes to learn the perturbation law of score-based diffusion models. Each forward trajectory is assigned a latent noise scale drawn from a distribution that is learned, and the score network is conditioned on this scale. The authors observe that denoising score matching contains an irreducible term depending on the perturbation law. They propose a corrected criterion that subtracts an estimate of this term obtained from a cross-fitted denoiser. The perturbation law is given a normalized Dirichlet-process prior and learned by generalized Bayesian inference with this criterion. The theory covers consistency of the corrected criterion, concentration of the generalized posterior around the perturbation laws that minimize the risk of the trained score model, a finite-sample bound for a linear score class, and a path-space bound linking score risk to reverse-process error. Experiments on a synthetic model, CIFAR-10, and HRRR weather fields report that the preferred perturbation law varies across tasks and training budgets.

Critical strengths

The paper studies an interesting problem, learning the perturbation law of diffusion models, and proposes a principled formulation based on a latent noise scale with a nonparametric prior. It provides solid theoretical guarantees, covering consistency of the corrected criterion, posterior concentration, and reverse-process error. The paper is well written, and the experiments are clearly designed and presented.

Critical weaknesses

First, the model reduces to choosing noise schedules. The introduction suggests learning general forward perturbation laws, but the construction is a random scalar diffusion coefficient with a λ\lambda-conditioned score. In the VP setting, Xtλ/at=X0+σZX^\lambda_t/a_t = X_0 + \sigma Z with σ2=λvt/at2\sigma^2 = \lambda v_t/a_t^2. The conditional score therefore depends on (t,λ)(t,\lambda) only through the effective noise level σ\sigma, up to the known scaling ata_t. Learning GG amounts to choosing the training distribution of noise levels, the loss weighting, and a mixture of sampling schedules, which existing methods already address (e.g., EDM's training noise distribution, learned schedules as in VDM, and sampling-schedule optimization). For the same reason, the Student-tt baselines are not heavy-tailed diffusions in the sense of prior work, since the noise is Gaussian conditional on λ\lambda. Baselines that directly tune the noise-level distribution and schedule are missing.

Second, the reverse-time analysis is not rigorous. Introducing the random scale Λ\Lambda requires care that the paper does not take:

  • Assumption E.1 only posits weak existence of the reverse SDEs for GG-a.e. λ\lambda. Without uniqueness in law and measurable dependence on λ\lambda, neither the identification of the reverse process with the time-reversed forward process nor the joint path measure G(dλ) Qλrev(dω)G(d\lambda)\,Q^{\rm rev}_\lambda(d\omega) used in Theorem 3.5 is justified.
  • The Girsanov step assumes a Novikov condition for every λ\lambda. This is unverifiable for learned scores, and existing analyses (Chen et al., 2023; Benton et al., 2024) need only L2L^2 score error and finite second moments.
  • The admissible class G\mathcal G is never defined, although Lemma E.2 requires λ≥λmin⁡>0\lambda \ge \lambda_{\min} > 0.
  • The extension to τ=0\tau = 0 at the end of App. E is not justified, since the score is singular as t→0t\to0.

Third, the corrected criterion does not isolate the score risk. The plug-in correction equals the DSM loss of the score induced by the nuisance denoiser through Tweedie's formula, C^τ(G,r^)=LDSM,τ(G,sr^)\hat C_\tau(G,\hat r) = L_{\rm DSM,\tau}(G, s_{\hat r}). The criterion therefore estimates Rscore(G,s^)−Rscore(G,sr^)R_{\rm score}(G,\hat s) - R_{\rm score}(G, s_{\hat r}), which equals the score risk only when the denoiser is far more accurate than the score model. Theorems 3.1–3.2 assume this through uniform denoiser consistency, which conflicts with the finite-resource premise of the paper. On CIFAR-10, the score model is itself an x0x_0-prediction denoiser trained for up to 100k updates, the same budget as the nuisance denoisers. The residual GG-dependent bias is not analyzed.

Fourth, the theoretical results are weak:

  • Theorem 3.2 assumes that the trained score risks concentrate uniformly around their expectation, which is close to the conclusion. Its target corresponds to the fold size rather than the final training setting.
  • Theorem 3.4 is a standard bound for least squares with fixed features in the underparameterized regime. It is stated through abstract GG-dependent quantities (Eapp(G)E_{\rm app}(G), ℓG\ell_G, Cτ(G)C_\tau(G)) rather than explicit problem parameters, and it is only an upper bound. It does not cover overparameterized networks, for which empirical DSM minimizers approach the empirical score and early stopping governs the risk. The experiments vary resources through training updates, which the theory does not model.

Fifth, the method is computationally impractical, and the theory does not describe the algorithm that is run:

  • Each evaluation of the criterion requires training cross-fitted score models and denoisers under GG. The paper instead alternates updates of a shared score model and the variational parameters of GG, a bilevel heuristic that is not analyzed and to which Theorem 3.3 does not apply.
  • Costs are multiplied by cross-fitting, the additional nuisance denoisers, and retraining under the frozen law. None of these costs is reported, and the baselines are not compared at matched compute.
  • The nonparametric aspect is nominal: the stick-breaking is truncated at K=4K = 4 atoms on [0.5,2][0.5, 2].

Finally, results are reported only in corrected score risk, without error bars or generation metrics. On CIFAR-10 at 100k updates, all values lie between 0.0002 and 0.0016, which makes the reported ordering hard to interpret.

Questions for authors

Minor comments and questions

  • The symbol TT denotes both the terminal time and the random diffusion time T∼ρT\sim\rho (Sec. 3.3, App. A.1); distinct symbols would help.
  • The process is no longer variance preserving for λ≠1\lambda\neq1, since Var(Xtλ)=at2 Var(X0)+λvt\mathrm{Var}(X^\lambda_t) = a_t^2\,\mathrm{Var}(X_0) + \lambda v_t, which makes the name somewhat misleading.
  • The criterion uses the likelihood weighting λg(t)2\lambda g(t)^2 on [τ,T][\tau,T], whereas the score models are trained with an unweighted x0x_0-prediction loss and t∼Unif(0,1)t\sim\mathrm{Unif}(0,1) (App. G), which also covers t<τt<\tau. Could the authors comment on this mismatch?
  • The metric "dist" on G\mathcal G in Theorem 3.3 is not specified, and the prior-mass condition is stated only informally. It would help to make both precise in the main text.
  • The normalization requires λˉ(G~)<∞\bar\lambda(\tilde G)<\infty almost surely, which depends on the base measure HH; this condition should be stated.
  • For the Student-tt baselines, Λ∼InvGamma(ν/2,(ν−2)/2)\Lambda\sim\mathrm{InvGamma}(\nu/2,(\nu-2)/2) is not bounded away from zero, and Var(Λ)=∞\mathrm{Var}(\Lambda)=\infty for ν≤4\nu\le4, including ν=3\nu=3. These laws therefore do not satisfy the condition λ≥λmin⁡\lambda\ge\lambda_{\min} of Lemma E.2.
  • The main text uses the learning rate κn\kappa_n, while App. F reports both κ=2048\kappa = 2048 and a "temperature" η∈{0.25,1,4}\eta\in\{0.25,1,4\} described as preregistered. How do these relate?
  • The identity stλ=E[Ytλ∣Xtλ]s^\lambda_t = \mathbb E[Y^\lambda_t\mid X^\lambda_t] is attributed to Fisher's identity (App. A.2); it is more commonly referred to as the denoising score matching identity or Tweedie's formula.
  • "YOON et al., 2023" in the introduction should not be capitalized.
4 comments

Thank you for the careful review. We agree that the key questions are how our approach differs from existing noise-design methods, whether the corrected criterion can reliably compare perturbation laws, and how the theoretical guarantees relate to the implemented algorithm.

Our starting point is that the preferred perturbation law depends on the score learner and its training resources. This creates a statistical problem that is not resolved by allowing the noise distribution to vary: standard DSM values contain a law-dependent irreducible component and therefore need not rank perturbation laws by the score risk attained after training. Our contribution is to formulate this finite-resource target, construct a denoiser-based correction for cross-law comparison, and use the resulting criterion for BNP learning of trajectory-level scale distributions.

To W1 — Novelty relative to EDM/VDM:

We agree that adapting diffusion noise is not new. EDM examines noise-level sampling and weighting, while VDM learns a parameterized noise schedule. We address a different question: which perturbation law minimizes the expected score risk attained by a specified learner after finite training? Section 3.2 formalizes this through Gm,As⋆G_{m,\mathcal A_s}^{\star}. Changing GG changes the noisy inputs and score targets faced by that learner, so the best law can depend on model capacity and budget. Moreover, optimizing raw DSM across laws does not solve this selection problem: LDSM,τ(G,s)=Rscore,τ(G,s)+Cτ(G)\mathcal L_{\mathrm{DSM},\tau}(G,s)=\mathcal R_{\mathrm{score},\tau}(G,s)+C_\tau(G), with a law-dependent Cτ(G)C_\tau(G). Our correction addresses this comparison bias; BNP adapts the locations and masses of trajectory-level scales without fixing a short parametric scale family. In Figure 1, raw DSM reverses 29/36 pairwise exact-risk orderings, versus 0/36 with the analytic correction. The conditional VP transitions remain Gaussian; our contribution is a corrected criterion for learner-dependent law selection.

To W2 — Reverse SDE and path-space KL:

Because a scale Λ\Lambda is drawn once per trajectory, the XtX_t marginal need not be Markov. The reverse construction therefore conditions on Λ\Lambda, equivalently working on the augmented state (Xt,Λ)(X_t,\Lambda). We revised Theorem 3.5 and Appendix E to state the path-law requirements explicitly: the exact law is the time reversal of the forward process; the reverse equations are well posed with measurable scale-indexed kernels; terminal laws satisfy absolute continuity; and the Girsanov density is a true martingale with finite energy. Under these conditions and 0<τ<T0<\tau<T, the joint latent-scale/path laws satisfy KL⁡(PGrev∥QG,s^rev)=DT(G)+Rscore,τ(G,s^)\operatorname{KL}(\mathbf P_G^{\mathrm{rev}}\Vert\mathbf Q_{G,\widehat s}^{\mathrm{rev}})=\mathcal D_T(G)+\mathcal R_{\mathrm{score},\tau}(G,\widehat s).

The proof separates reverse-drift error from terminal initialization mismatch DT(G)\mathcal D_T(G). Appendix E supplies sufficient conditions for compact positive-scale VP diffusion. The endpoint τ=0\tau=0 is treated under the additional regularity specified in Appendix E.

To W3 — Estimated nuisance correction and cross-law ranking:

We already provide an explicit analysis of nuisance estimation error and its effect on cross-law ranking in Appendix B. Specifically, Eq. (7) shows that, conditional on the complementary fitting data D−kD_{-k}, the expected held-out corrected criterion equals the fitted score risk minus the nonnegative weighted nuisance excess risk. Thus, cross-law ranking depends on the difference in nuisance excess risks relative to the underlying score-risk gap.

Lemma B.1 further bounds the estimation error through held-out fluctuations, nuisance error, and trained-score fluctuations. A true score-risk gap exceeding twice the corresponding uniform error bound preserves pairwise ordering on the event where that bound holds.

We additionally examined this mechanism using the nine perturbation laws from Figure 1. With 16,384 nuisance samples and 20 paired seeds, replacing the original ridge denoiser with a small MLP trained for up to 1600 epochs reduced ranking reversals exceeding two evaluation standard errors from 21/36 to 0/36, while the maximum cross-law nuisance-excess gap decreased from 0.742 to 0.038. The remaining single sign reversal was within evaluation uncertainty.

This confirms the predicted relationship between nuisance estimation accuracy and cross-law ranking in the controlled synthetic setting.

To W4 — Statistical guarantees for finite-resource law learning:

Our theory establishes how perturbation laws can be compared and learned according to the score risk achieved under finite training resources. Theorems 3.1–3.2 justify the corrected-risk criterion under their uniform estimation and concentration assumptions, extending from profiled score risk to the risk of a specified score-training algorithm. Building on this foundation, Theorem 3.3 establishes concentration of the ideal generalized posterior near the finite-resource optimal perturbation laws under explicit risk-gap and prior-mass conditions.

Theorem 3.4 then explains why the choice of perturbation law matters for finite-sample learning. For fixed-feature weighted least squares, its risk bound separates approximation error, DSM target variance Cτ(G)C_\tau(G), and weighted leverage, whose values can depend on GG. This identifies how the perturbation law influences finite-sample score estimation. Here mm counts training samples, while Table 2 varies neural optimization updates.

This connects risk estimation, Bayesian law selection, and finite-sample score learnability within one statistical framework.

To W5 — Relation to the implemented BNP algorithm:

Our implementation uses a shared score network and finite-KK variational inference to make Bayesian perturbation-law learning computationally tractable. Rather than repeatedly retraining the score model for every candidate GG, we alternate updates of the shared score network and the variational distribution qϕq_\phi over normalized scale laws (Section 2.4; Appendix G).

The procedure has two stages. During law adaptation, two independently initialized nuisance denoisers, trained for 100k updates under a fixed reference proposal, provide a shared correction across candidate laws. After adaptation, we freeze qϕq_\phi, discard the adaptation-stage score model, and train a fresh network under the learned law distribution (Section 4.2). This separates learning the perturbation law from evaluating its effect on final score training.

Theorem 3.3 studies the ideal generalized posterior; our finite-KK variational procedure implements law adaptation using a corrected-risk surrogate. Appendix F examines sensitivity to K∈{2,4,8}K\in\{2,4,8\}, α∈{0.1,1,10}\alpha\in\{0.1,1,10\}, and η∈{0.25,1,4}\eta\in\{0.25,1,4\}.

To W6–W7 — Empirical evidence and computational comparison:

Our experiments demonstrate both the importance of perturbation-law selection and the ability of BNP to adapt scale distributions to different learning problems. Figure 1 validates the DSM-ranking correction against exact finite-sample score risk, while Table 1 shows that BNP learns distinct scale distributions in Gaussian and non-Gaussian reference regimes, reflecting their different score-learning requirements.

Table 2 further demonstrates the practical value of BNP adaptation. On CIFAR-10, BNP achieves the lowest estimated corrected risk at 25k updates (0.0421 vs. Gaussian's 0.0504). On HRRR VIL, BNP improves over Gaussian at 100k updates (0.0544 vs. 0.0889), and on HRRR w20 it achieves the lowest reported risk at 25k (0.8411). At CIFAR-10 100k, Student-tt with ν=3\nu=3 performs best (0.0002 vs. BNP's 0.0013), illustrating how the preferred perturbation law changes with the training budget. These results highlight BNP's ability to learn competitive perturbation laws across different score-learning settings.

Regarding computational cost, we match final score-training budgets across methods. BNP additionally performs nuisance fitting and law adaptation. On CIFAR-10 at 100k updates, BNP adaptation required approximately 8.743 hours, compared with 2.323 hours for the Gaussian comparator stage on the same RTX 5090.

To Minor comments (i)–(ix):

(i) Fixed terminal time TT differs from sampled T\mathsf T. (ii) Conditional VP variance is multiplied by λ\lambda, with one scale retained per trajectory. (iii) Training uses unweighted x0x_0 prediction and uniform time sampling; evaluation uses the likelihood-weighted corrected score-risk target, with qeval(t)∝1/tq_{\rm eval}(t)\propto 1/t on [τ,T][\tau,T] and its importance weights. (iv) Theorem 3.3 explicitly uses bounded-Lipschitz distance and near-optimal prior mass. (v) Mean normalization is well defined under the finite positive raw-atom support. (vi) The Student-tt inverse-gamma mixture is an unbounded-scale fixed-law baseline, outside the compact-scale uniform terminal bound. (vii) Synthetic κ=2048η\kappa=2048\eta is distinct from the theoretical sequence κn\kappa_n. (viii) The conditional-expectation score identity follows the standard denoising/Tweedie identity. (ix) The capitalization originated from the NeurIPS proceedings BibTeX author field (YOON, EUN BI); we standardized the citation to Yoon et al., 2023 in the revised manuscript.

Thank you for the detailed rebuttal and additional clarifications. The responses have addressed most of my concerns, particularly regarding the position of the contribution, assumptions, and theoretical analysis. I also appreciate the additional experimental evidence.

Some limitations remain, particularly the strong regularity assumptions (I think can be further weakened) and the scope of the statistical analysis guarantees, but I recognize that these may be difficult to address within the current framework.

I have updated my assessment accordingly.

Thank you for your thoughtful follow-up and for reconsidering your assessment. We are glad that our clarifications and additional evidence have addressed your main concerns.

We also appreciate your suggestions on relaxing the regularity assumptions and extending the statistical analysis, which provide valuable directions for further theoretical development.

Thank you again for your careful and constructive evaluation.

Reviewer_deLxOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies the finite-resource learning problem of the “perturbation law” in diffusion models. The authors identify two difficulties. First, the optimal perturbation law should depend on the specific score learner and training budget. Second, the standard DSM objective is not directly comparable across different perturbation laws, because the population DSM loss can be decomposed into the score risk and an irreducible variance term C_T(G) that depends on G. To address this, the authors introduce a trajectory-level latent scale Λ ∼ G and place a Dirichlet process prior on G. Using a Bayes denoiser and cross-fitting, they estimate and subtract C_T(G) to construct a “corrected DSM risk,” and then use generalized Bayes to update the posterior of G. The theoretical part provides consistency of the corrected risk, risk consistency of the cross-fitted algorithm, posterior concentration, finite-sample score-risk bounds, and a connection between reverse-process KL and score risk. Experiments include one-dimensional synthetic experiments and comparisons on CIFAR-10 and HRRR under two training budgets.

Critical strengths

The problem formulation and the correction criterion have substantial value. The authors define “selecting a perturbation law under finite resources” as algorithm-dependent score risk, point out that raw DSM is not comparable across G, and then use a Bayes denoiser to subtract C_t(G). In the synthetic experiments, the ordering of raw DSM severely deviates from that of exact score risk, while the corrected criterion closely tracks exact risk. This is one of the strongest mechanistic pieces of evidence in the paper and directly supports the method’s motivation.

The theoretical chain is relatively complete. From corrected-risk consistency, cross-fitted algorithm risk, and posterior concentration to finite-sample bounds and reverse KL bounds, the paper attempts to incorporate “learning the perturbation law” into an analyzable statistical framework. Even though some assumptions are strong, such end-to-end theoretical treatment is uncommon in related work on learnable forward/noise schedules.

Critical weaknesses

The empirical advantage on real data is mixed, and generation-quality metrics are missing. In Table 2, on CIFAR-10 at 100k, the corrected risk of Student-t (v = 3) is 0.0002, clearly better than BNP’s 0.0013; on HRRR VIL at 25k, Gaussian is 0.5381, better than BNP’s 0.5618. BNP is best only in some settings. The paper’s claim that “preferences vary with task and budget” is acceptable, but to support the claim that “BNP is an effective adaptive perturbation learner,” it needs to demonstrate that improvements in corrected risk translate into generation/prediction metrics such as FID/IS, likelihood, HRRR downstream forecast error, or CRPS; otherwise, the method-selection criterion and the evaluation criterion are the same, creating a risk of circular reasoning.

The nonparametric implementation, theoretical claims, and computational cost/fairness need clarification. The theory uses an infinite Dirichlet process, but the experiments use a K = 4 truncation, restrict atoms to [0.5, 2], and normalize the mean. This may limit its ability to represent heavy-tailed or more complex optimal laws; the failure of CIFAR-10 at 100k to match Student-t (v = 3) may be related to this. In addition, BNP adaptation requires alternating updates, cross-fitting, training a nuisance denoiser, and finally freezing the perturbation law and retraining the score model from scratch. The main text does not fully explain whether fixed-law baselines are compared under the same total computational budget and the same hyperparameter search.

Questions for authors

(1) Can you add generation/prediction quality metrics on CIFAR-10 and HRRR (such as FID, IS, likelihood, CRPS, or downstream forecast error) to compare Gaussian, Student-t, and BNP under 25k/100k budgets? Do improvements in corrected score risk translate into actual generation gains? (2) The experiments use a truncated DP with K = 4 and atom support [0.5, 2]. Please provide sensitivity analyses for K, α, the support range, and the generalized-Bayes learning rate κ. How should the infinite-DP posterior concentration result in the theory be understood under the truncated implementation? (3) Please clarify how the total computational cost of BNP adaptation, cross-fitting nuisance denoiser training, and final retraining matches the baseline tuning budget. Have fixed-law baselines undergone the same degree of hyperparameter search? (4) On CIFAR-10 at 100k, BNP’s corrected risk is 0.0013, while Student-t (v = 3) is 0.0002. Do the authors consider this due to representational limitations, optimization difficulty, validation selection bias, or that BNP should not be optimal in this setting? Can you show where the posterior mass concentrates over scales and whether it is close to a Student-t scale mixture? (5) The correction term depends on the Bayes denoiser estimate. On high-dimensional real data, how much does the nuisance denoiser error affect the ordering across G? Is there a sensitivity experiment or a theoretical error-propagation bound? (6) Compared with existing work on learnable forward processes/noise schedules/heavy-tailed diffusion (e.g., Bartosh et al., Pandey et al., Shariatian et al.), what are the substantive differences and advantages of this paper? Can you provide direct experimental comparisons?

6 comments

Thank you for recognizing the finite-resource formulation, DSM correction and theoretical treatment. We agree that the central empirical question is what the observed corrected-risk differences establish about perturbation-law learning, and where their implications end.

For a specified score learner and training budget, Section 3.2 defines the perturbation-law target through the expected score risk attained after training. Because the irreducible DSM component depends on the perturbation law (Section 2.3), ordinary DSM values need not compare candidates by this target. Our correction addresses that obstacle, while BNP provides a mechanism for adapting the distribution of trajectory-level noise scales. We address the empirical evidence, implementation, and statistical questions below.

To W1 — Empirical evidence and independent validation:

Our experiments are designed to examine how the perturbation law preferred for score estimation depends on the task and training resources. Table 2 provides evidence for this dependence. On CIFAR-10, BNP achieves lower estimated corrected risk than Gaussian at 25k final updates (0.0421 vs. 0.0504), while Student-tt with ν=3\nu=3 performs best at 100k (0.0002 vs. BNP's 0.0013). On HRRR VIL, Gaussian performs better at 25k (0.5381 vs. 0.5618), whereas BNP achieves lower risk at 100k (0.0544 vs. 0.0889). These results illustrate why perturbation-law selection should account for both the learning problem and the training budget.

Importantly, our correction is also evaluated against an independently computable reference. In Figure 1, raw DSM reverses 29/36 pairwise rankings relative to exact finite-sample score risk, while the analytically corrected criterion reverses none. This directly validates the cross-law correction mechanism in the controlled synthetic setting. On real data, CIFAR-10 law selection uses held-out validation observations and its final corrected-risk assessment uses the untouched test set; HRRR reports corrected validation risk.

The experiments therefore provide direct evidence for the finite-resource score-learning objective and the need for cross-law correction, which are the central focus of our work.

To Q1 — Generation and prediction quality:

The central question of our paper is how to select and adapt perturbation laws for score estimation under finite training resources. Corrected score risk directly measures this objective, whereas FID and downstream prediction metrics assess different aspects of model performance. Theorem 3.5 further connects score-estimation risk to the reverse generative process through a path-space KL identity that also accounts for terminal initialization mismatch. Our experiments are designed to evaluate this learner-dependent statistical objective across tasks and training budgets.

To Q4 — CIFAR-10 at 100k and resource-dependent perturbation preferences:

The CIFAR-10 results illustrate a central finding of our paper: the preferred perturbation law changes with the available score-training budget. BNP achieves the lowest corrected risk at 25k updates (0.0421), whereas Student-tt with ν=3\nu=3 performs best at 100k (0.0002). More importantly, the preference reverses even within the Student-tt family: ν=10\nu=10 performs best among the Student-tt candidates at 25k, while ν=3\nu=3 becomes best at 100k.

Figure 2 shows that at 100k updates, the fitted BNP posterior-predictive scale distribution remains concentrated near unit scale, whereas the Student-tt mixture with ν=3\nu=3 is considerably more dispersed. This makes their different scale allocations visible alongside the reported risk ordering. The effectiveness of a perturbation law is tied to the score learner and its training resources, motivating our finite-resource formulation and adaptive law-learning framework.

To W2 / Q2 — Finite BNP implementation and sensitivity:

The model places a normalized nonparametric prior on the law of a trajectory-level scale. Theorem 3.3 concerns the ideal generalized posterior under an explicit risk gap, uniform risk-estimation error and near-optimal prior mass, whereas the implemented algorithm uses a finite-KK variational distribution qϕq_\phi. The experimental implementation uses a truncated stick-breaking qϕq_\phi, K=4K=4, α=1\alpha=1, generalized-Bayes rate κ=2048\kappa=2048, and mean-normalized raw atoms from H=Uniform⁡(0.5,2)H=\operatorname{Uniform}(0.5,2) (Appendix F).

Appendix F reports targeted synthetic one-at-a-time checks for K∈{2,4,8}K\in\{2,4,8\}, α∈{0.1,1,10}\alpha\in\{0.1,1,10\} and rate multiplier η∈{0.25,1,4}\eta\in\{0.25,1,4\}, where κ=2048η\kappa=2048\eta. With raw support expanded to [0.25,4][0.25,4], the expected corrected surrogate risk in the non-Gaussian-reference regime changed from 0.8281 to 0.3954, while expected within-law scale variance changed from approximately 0.457 to 2.196. In the Gaussian-reference control, corrected surrogate risk changed from 0.6929 to 0.6953. The widened interval also changes the uniform base measure, so this experiment measures the sensitivity of the accessible law family and prior specification.

To Q3 — Compute and baseline tuning:

The experiments match score-network architecture, optimizer and final fresh training updates within each task (25k/100k); BNP adds a law-adaptation stage. BNP first fits the shared nuisance denoisers and adapts qϕq_\phi, then discards the adaptation-stage score network and trains a fresh network under the frozen qϕq_\phi (Section 4.2; Appendix G). Fixed Student-tt candidates are chosen from the validation grid ν∈{3,5,10}\nu\in\{3,5,10\}. On the same RTX 5090, CIFAR-10/100k BNP law adaptation required approximately 8.743 hours versus 2.323 hours for the Gaussian comparator stage.

To Q5 — Nuisance accuracy and ranking:

Appendix B.2 shows that the expected held-out corrected loss, conditional on the fitted score and nuisance, equals fitted-score risk minus nonnegative weighted nuisance excess risk. It is therefore the difference in nuisance excess risk across laws, relative to the actual score-risk gap, that controls bias-induced ranking errors; using a shared denoiser and cross-fitting does not automatically remove this bias.

In an additional one-dimensional synthetic experiment using 16,384 nuisance samples and 20 paired seeds, replacing the original ridge denoiser with a small MLP trained for up to 1600 epochs reduced ranking reversals exceeding two evaluation standard errors from 21/36 to 0/36. The maximum cross-law nuisance-excess gap fell from 0.742 to 0.038. One remaining sign reversal was within two evaluation standard errors. This directly illustrates how improved nuisance estimation protects law ranking in the controlled setting.

To Q6 — Relation to learned forward processes and heavy-tailed methods:

Our work addresses a different statistical question from existing approaches to forward-process and noise-schedule design: how to learn a perturbation law for a given score learner under finite training resources. We adopt a trajectory-level latent-scale diffusion to preserve the conditional Gaussian structure needed for score learning and reverse-diffusion analysis, while allowing BNP to adapt the distribution of perturbation scales.

Our main contributions are the learner- and budget-dependent score-risk target (Section 3.2), the law-dependent DSM correction for comparing candidate laws (Section 2.3), and generalized Bayesian adaptation over scale distributions (Section 2.4). This provides a statistical framework for learning perturbation laws rather than prescribing a fixed perturbation family. As discussed in Section 5, the same research direction also motivates exploring richer perturbation families and learning procedures.

Dear Reviewer deLx,

We are writing to ask whether our rebuttal and revised manuscript have addressed your main concerns. We would be happy to clarify or discuss any remaining issues.

Thank you again for your time and consideration.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.