acceptodds
Under review as a conference paper at ICLR 2027

Diffusion Models with Heavy-Tailed Targets: Score Estimation and Sampling Guarantees

10 views

Abstract

Score-based diffusion models have emerged as a powerful framework for generative modeling, with score estimation as a central statistical bottleneck. However, existing theoretical guarantees largely focus on light-tailed distributions or impose restrictive assumptions such as compact support, limiting their applicability to heavy-tailed data encountered in practice. In this work, we study standard score-based diffusion models for heavy-tailed distributions in a Sobolev class with smoothness , covering both exponential and polynomial tail decay characterized by a tail parameter . Based on kernel density estimation, we derive sharp minimax rates for score estimation, revealing a qualitative distinction between the two regimes: under exponential tails, the estimation rate matches the light-tailed benchmark up to polylogarithmic factors, whereas under polynomial tails it explicitly depends on the tail parameter . We further establish the convergence guarantees for the associated continuous-time reverse diffusion dynamics. For exponential tails and , the generated distribution converges in total variation at the rate up to logarithmic factors, matching the light-tailed benchmark. For polynomial tails, we obtain a -dependent total variation bound, while leaving its optimality open. These results characterize the statistical limits of score estimation and the resulting sampling accuracy for heavy-tailed targets, extending diffusion theory beyond the light-tailed setting.

Primary area:
learning theory
Keywords:
Score-based diffusion models, Heavy-tailed distributions, Score estimation, Minimax rates, Kernel density estimation

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_roTMOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

The paper studies score estimation via a thresholded Gaussian KDE, together with total-variation sampling guarantees for the continuous-time reverse SDE, for Sobolev-smooth targets with exponential or polynomial tail decay. Minimax lower bounds for score estimation are also derived in both regimes, matching the upper bounds up to logarithmic factors over restricted ranges of diffusion time.

Critical strengths
  1. The problem is well-motivated. Nearly all existing minimax theory for diffusion models assumes sub-Gaussian tails, compact support, or a density lower bound. This paper is a clean first step toward heavy tails within the standard Gaussian pipeline.
  2. The score-estimation results are sharp in a meaningful sense. Upper and lower bounds match up to polylog factors in both regimes. The lower-bound construction for polynomial tails is the most original technical part.
Critical weaknesses
  1. The polynomial-tail sampling rate seems to be strictly suboptimal, not just of unknown optimality. Any sampler's output law is a density estimator, so classical L1L^1 density estimation gives a benchmark. A second-order KDE with the standard truncation argument works as follows. Bound the L1L^1 error on a ball of radius RR by Rd/2n−β/(2β+d)R^{d/2} n^{−\beta/(2\beta+d)}, add the tail mass R−(γ+1)R^{−(\gamma+1)}, and balance the two. We have TV≲n−2β(γ+1)/((2β+d)(d+2γ+2))TV \lesssim n^{−2\beta(\gamma+1)/((2\beta+d)(d+2\gamma+2))}. That exponent's denominator, (2β+d)(d+2γ+2)(2\beta+d)(d+2\gamma+2), is smaller than Theorem 4's, 4β(d+γ+1)+d(d+2γ+2)4\beta(d+\gamma+1) + d(d+2\gamma+2), so the naive KDE rate is strictly faster for every γ\gamma. This matters for the paper's framing. It suggests the diffusion pipeline, or at least this analysis of it, loses something under heavy tails. Comparing against the KDE baseline would sharpen the discussion.
Questions for authors
  1. There is no discretization analysis in this work. The continuous-time reverse process is usually discretized to perform generative sampling. Would the authors provide the discretization error analysis or explain why such an analysis is not necessary?
  2. The forward process dXt=dBtdX_t=dB_t seems too simple. In practice, people would choose non-constant or even time-dependent drift and diffusion coefficients. Do the theoretical results extend to such general settings? If so, could the authors discuss the required assumptions and any additional technical challenges?
4 comments
AuthorsOpenReview ↗

Reply to Reviewer roTM Part 1

We thank the reviewer for the feedback. Please find our point by point response below.

  1. The polynomial-tail sampling rate seems to be strictly suboptimal, not just of unknown optimality. Any sampler's output law is a density estimator, so classical $L^1$ density estimation gives a benchmark. A second-order KDE with the standard truncation argument works as follows. Bound the $L^1$ error on a ball of radius $R$ by $R^{d/2} n^{−\beta/(2\beta+d)}$, add the tail mass $R^{−(\gamma+1)}$, and balance the two. We have $TV \lesssim n^{−2\beta(\gamma+1)/((2\beta+d)(d+2\gamma+2))}$. That exponent's denominator, $(2\beta+d)(d+2\gamma+2)$, is smaller than Theorem 4's, $4\beta(d+\gamma+1) + d(d+2\gamma+2)$, so the naive KDE rate is strictly faster for every $\gamma$. This matters for the paper's framing. It suggests the diffusion pipeline, or at least this analysis of it, loses something under heavy tails. Comparing against the KDE baseline would sharpen the discussion.

We agree that the proposed direct KDE-based distribution estimator achieves a faster TV convergence rate than the upper bound in Theorem 4, showing that our bound is not minimax-sharp as a general distribution estimation guarantee. However, comparing two upper bounds does not establish which sampling procedure actually converges faster. In particular, although the direct KDE estimator admits a stronger theoretical guarantee, Theorem 4 only provides an upper bound for our reverse-diffusion sampler driven by a thresholded KDE score, rather than a matching characterization of its actual convergence rate. Therefore, this comparison alone does not establish that our proposed diffusion sampler is statistically less efficient.

Moreover, the same KDE rate can be achieved using the empirical Gaussian KDE p^t(x)=1n∑i=1nφt(x−Xi)\hat p_t(x)=\frac{1}{n}\sum_{i=1}^n\varphi_t(x-X_i), which admits an exact reverse-diffusion representation through its unthresholded score when initialized from the corresponding smoothed empirical distribution. Thus, the faster KDE rate does not imply an intrinsic limitation of Gaussian diffusion under heavy tails. Rather, the gap may arise from our score-to-TV analysis, the particular thresholded-score construction, or both.

As outlined in the Introduction, our primary objective is to characterize the minimax-optimal rates for score estimation under heavy-tailed distributions and investigate how these estimation errors propagate to sampling accuracy. Our thresholded score estimator is designed to control errors in low-density regions and achieves the minimax-optimal score-estimation rate established in Theorems 1 and 9. Theorem 4 then derives a TV sampling guarantee through Girsanov's theorem and Pinsker's inequality, which may not yield a sharp bound for the proposed sampler. We do not claim minimax optimality of this sampling guarantee, as acknowledged in the abstract and Discussion.

We appreciate the suggested KDE comparison and will incorporate it into the revised manuscript.

  1. There is no discretization analysis in this work. The continuous-time reverse process is usually discretized to perform generative sampling. Would the authors provide the discretization error analysis or explain why such an analysis is not necessary?

Our primary objective is to characterize the fundamental statistical limits of score estimation under heavy-tailed distributions and investigate how these estimation errors propagate to sampling accuracy. We therefore focus on the continuous-time reverse SDE, which allows us to isolate the statistical error from numerical discretization errors.

Discretization introduces a separate computational question beyond the statistical objectives of this work. While important for practical implementations, its analysis is not required to establish our minimax score-estimation rates or continuous-time sampling guarantees. As acknowledged in the Discussion, extending our results to discretized samplers is an interesting direction for future work.

AuthorsOpenReview ↗

Reply to Reviewer roTM Part 2

  1. The forward process $dX_t=dB_t$ seems too simple. In practice, people would choose non-constant or even time-dependent drift and diffusion coefficients. Do the theoretical results extend to such general settings? If so, could the authors discuss the required assumptions and any additional technical challenges?

Our choice of Brownian motion is deliberate, as our primary objective is to characterize the statistical effects of heavy-tailed targets on score estimation and sampling under Gaussian noising, rather than to investigate different forward dynamics.

Importantly, the Gaussian-noising structure underlying our analysis also applies to a broader class of forward processes, including commonly used variance-exploding and variance-preserving diffusions. More generally, consider the linear SDE dXt=a(t)Xtdt+b(t)dBtdX_t=a(t)X_tdt+b(t)dB_t, where a(t)a(t) and b(t)b(t) are deterministic scalar functions. Define

\qquad v_t=\int_0^t\frac{b(s)^2}{\alpha_s^2}ds.$$ Assume $a$ is locally integrable, $b/\alpha$ is locally square-integrable, and $v_t$ is strictly increasing. Then, for each $t$, $$X_t\overset{d}{=}\alpha_t(X_0+\sqrt{v_t}Z), \qquad Z\sim N(0,I_d),\quad Z\perp X_0.$$ Let $q_v=p_0*\varphi_v$, where $\varphi_v$ denotes the density of $\mathcal{N}(0,vI_d)$, and define $s_v=\nabla\log q_v$. Then, the density $p_t^X$ of $X_t$ and its associated score $s_t^X$ are given by $$p_t^X(x)=\alpha_t^{-d}q_{v_t}(x/\alpha_t),\quad s_t^X(x)=\alpha_t^{-1}s_{v_t}(x/\alpha_t).$$ Consequently, the transformed estimator $\hat s_t^X(x)=\alpha_t^{-1}\hat s_{v_t}(x/\alpha_t)$ has weighted squared score risk scaled by $\alpha_t^{-2}$. Our upper and minimax lower bounds therefore transfer with $t$ replaced by $v_t$, provided the transformed times and distribution classes satisfy the respective theorem assumptions. For sampling, the diffusion-weighted cumulative score error satisfies $$\int_{t_0}^{T}b(t)^2\lVert \hat s_t^X-s_t^X \rVert_{L^2(p_t^X)}^2\,dt =\int_{v_{t_0}}^{v_T} \lVert \hat s_v-s_v \rVert_{L^2(q_v)}^2\,dv.$$ This is precisely the score-error term appearing in the corresponding Girsanov bound. Extending the sampling guarantees further requires appropriate transformations of the reverse dynamics and initialization, as well as rescaling the stopped output for comparison with $p_0$. These adjustments must be taken into account when establishing sampling rates under general noise schedules. In contrast, more general SDEs with nonlinear drifts or state-dependent diffusion coefficients need not preserve the Gaussian-convolution structure. Extending our analysis to such settings would require additional estimates for the transition densities, their tail behavior, and the associated score functions, together with appropriate modifications to the reverse-time sampling analysis. We will clarify these distinctions in the revised manuscript.

My concerns are properly addressed. I raised the score.

Thank you for your positive feedback and for reconsidering your assessment.

Reviewer_vxS9OpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper studies score-based diffusion models for heavy-tailed distributions in a Sobolev class with smoothness parameter β>0\beta>0. The score function is estimated via kernel density estimation. The authors establish minimax rates for score estimation and provide convergence guarantees for the associated continuous-time reverse diffusion process.

Critical strengths
  1. A minimax optimal rate is established for score estimation under heavy-tailed distributions.
  2. Convergence guarantees for diffusion models are provided, showing how score estimation error propagates to the distribution error in sampling.
Critical weaknesses
  1. The authors state that the score estimation errors in Theorems 1 and 5 achieve the minimax optimal rate. I wonder whether the resulting convergence guarantee in Theorem 4 also achieves the minimax rate, in the sense of nonparametric distributional estimation.

  2. In Theorem 2 and Theorem 9, it appears that ∥s^t(x)−st(s)∥L2(pt)2=∫∥s^t(x)−st(s)∥2pt(x)dx\|\hat{s}_t(x)-s_t(s)\|_{L_2(p_t)}^2=\int \|\hat{s}_t(x)-s_t(s)\|^2p_t(x)\mathrm{d} x, but different notations are used. The authors may wish to clarify this or unify the notation.

  3. Most analyses are restricted to β∈(0,2]\beta\in(0,2]. Could the authors extend the minimax rates to the case β>2\beta > 2?

  4. A kernel density estimator is used, which does not align with practical implementations where neural networks are employed. While I understand that the authors use this to simplify the analysis, it would be helpful to discuss the implications of this gap.

  5. The authors only consider continuous-time reverse diffusion processes. It would be better to extend the analysis to discrete-time reverse processes, which are the ones actually used in practice.

Questions for authors

see weakness.

3 comments

We thank the reviewer for the positive feedback. Please find our point by point response below.

  1. The authors state that the score estimation errors in Theorems 1 and 5 achieve the minimax optimal rate. I wonder whether the resulting convergence guarantee in Theorem 4 also achieves the minimax rate, in the sense of nonparametric distributional estimation.

As outlined in the Introduction, our goal is to investigate how score estimation rates translate into sampling error bounds. Theorem 4 establishes such a bound in TV distance under polynomial tails, while its minimax optimality remains open, as noted in the abstract and Discussion.

In particular, our sampling analysis relies on Girsanov's theorem and Pinsker's inequality, which may not yield a sharp rate. Establishing minimax optimality is challenging, especially in the low-noise regime, where our score-estimation lower bounds do not apply and our sampling analysis relies on early stopping. As discussed in Section 5, a sharp characterization of the sampling rate would require a separate analysis under the TV loss, potentially involving sharper error-propagation bounds or new minimax lower-bound techniques. We leave this challenging problem for future work.

  1. In Theorem 2 and Theorem 9, it appears that $|\hat{s}_t(x)-s_t(s)|_{L_2(p_t)}^2=\int |\hat{s}_t(x)-s_t(s)|^2p_t(x)\mathrm{d} x$, but different notations are used. The authors may wish to clarify this or unify the notation.

As defined in the Notation paragraph at the end of the Introduction, $ \lVert f \rVert_{L^2(q)}^2=\int_{\mathbb R^d}|f(x)|^2q(x)dx$. Thus, the expressions in Theorems 2 and 9 are equivalent, with the latter written explicitly in integral form.

  1. Most analyses are restricted to $\beta\in(0,2]$. Could the authors extend the minimax rates to the case $\beta > 2$?

We would like to clarify that our minimax score-estimation results already accommodate general Sobolev smoothness β>0\beta>0. Specifically, Theorems 1 and 5 establish upper bounds for score estimation, while Theorems 9 and 10 provide matching minimax lower bounds over the specified diffusion-time ranges. The restriction β∈(0,2]\beta\in(0,2] applies only to the sampling guarantees in Theorems 4 and 8 and the associated early-stopping bias bounds.

This restriction arises from Gaussian smoothing, whose approximation bias generally saturates at order tt even for β>2\beta>2. Exploiting higher-order smoothness to obtain improved sampling rates would therefore require additional techniques, such as higher-order bias correction, which are beyond the scope of our current analysis.

  1. A kernel density estimator is used, which does not align with practical implementations where neural networks are employed. While I understand that the authors use this to simplify the analysis, it would be helpful to discuss the implications of this gap.

Our use of KDE is a deliberate choice motivated by the primary objective of this work: characterizing the fundamental statistical limits of score estimation under heavy-tailed distributions. Under the Gaussian forward process, the smoothed density admits an explicit convolution representation, making KDE a natural choice for constructing score estimators and establishing sharp statistical guarantees.

Importantly, our minimax lower bounds are estimator-independent and therefore apply to neural network-based score estimators as well. Together with the matching KDE upper bounds, these results characterize the fundamental statistical difficulty of score estimation, independently of any particular neural network architecture. While establishing analogous upper bounds for trained neural networks would require additional analysis of their approximation and estimation errors, our results provide a theoretical benchmark for their statistical performance under heavy-tailed distributions.

  1. The authors only consider continuous-time reverse diffusion processes. It would be better to extend the analysis to discrete-time reverse processes, which are the ones actually used in practice.

We thank the reviewer for this suggestion. Our primary objective is to characterize the statistical limits of score estimation under heavy-tailed distributions and investigate how these errors propagate to sampling accuracy. We therefore focus on continuous-time reverse diffusion to isolate the statistical error from the additional numerical error introduced by discretization.

While discretization analysis is important for practical implementations, it addresses a separate computational aspect and is not essential to the statistical questions studied here. As acknowledged in the Discussion, extending our guarantees to discrete-time samplers is an interesting direction for future work.

Thank you for your response and clarifications. I now understand the result in Theroem 4, the notation, and the range of paremter β\beta. I have no further questions and I would like to keep my score.

Thank you for your feedback and for confirming that our clarifications have addressed your concerns.

Reviewer_MftoOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies score estimation and sampling for Gaussian diffusion models under polynomial and exponential tail assumptions. The authors construct a thresholded score estimator using the empirical Gaussian KDE and establish upper bounds together with matching minimax lower bounds over specified diffusion-time windows. Polynomial tails introduce an explicit dependence on the tail parameter, while exponential tails retain the light-tail rate up to logarithmic factors. The paper then derives total-variation guarantees for continuous-time reverse diffusion with Gaussian initialization and early stopping.

Critical strengths

a) The distinction between polynomial and exponential tails is a meaningful statistical contribution in my opinion. The matching score-estimation lower bound shows that the polynomial-tail penalty reflects a difficulty inherent to the distribution class, rather than merely a weakness of the proposed estimator.

b) The polynomial-tail lower-bound construction is particularly interesting. The authors place perturbations in an expanding tail region, where observations are scarce, while maintaining sufficient separation between the corresponding scores./

Critical weaknesses

My main concerns are about the sampling comparison and the positioning relative to prior work. I would phrase them as questions:

a) Could the authors compare Theorem 4 with a direct analysis of the empirical Gaussian KDE? Its unthresholded score defines a reverse SDE that reproduces the KDE when initialized from the empirical mixture at time (T). Replacing this initialization by N(0,TId)N(0,TI_d) costs only O(T−1/2)O(T^{-1/2}) in expected TV under the stated assumptions.

b) Appendix A.1 states that the Rényi-entropy approach used by Zhang et al. is unavailable in this setting. However, the polynomial envelope still makes (\int p_t^\alpha) finite for suitable (\alpha<1), while the exponential envelope gives finiteness for every (\alpha>0). Could the authors clarify which step fails and which estimates require new arguments?

Questions for authors

see above?

2 comments

We thank the reviewer for the positive feedback. Please find our point by point response below.

  1. Could the authors compare Theorem 4 with a direct analysis of the empirical Gaussian KDE? Its unthresholded score defines a reverse SDE that reproduces the KDE when initialized from the empirical mixture at time (T). Replacing this initialization by $N(0,TI_d)$ costs only $O(T^{-1/2})$ in expected TV under the stated assumptions.

We agree that the reverse SDE driven by the unthresholded empirical KDE score reproduces the KDE distribution when initialized from the corresponding empirical mixture, and that replacing this initialization with a Gaussian introduces an additional expected TV error of order T−1/2T^{-1/2}. A direct KDE analysis can indeed yield a sharper TV convergence rate than the bound in Theorem 4.

However, our primary objective is to characterize the minimax-optimal rates for score estimation under heavy-tailed distributions and investigate how these errors propagate to sampling accuracy. Importantly, the unthresholded KDE score may have infinite weighted L2(pt)L^2(p_t) estimation error under polynomial tails, which motivates our thresholded construction. Our estimator achieves the minimax-optimal score estimation rate established in Theorems 1 and 9, while Theorem 4 quantifies the resulting sampling error through Girsanov's theorem and Pinsker's inequality.

The KDE comparison highlights that this score-to-TV conversion may not be sharp, but does not affect our minimax score estimation results. As discussed in Section 5, we do not claim minimax optimality of Theorem 4. We will include this comparison and further clarify the distinction in the revised manuscript.

  1. Appendix A.1 states that the Rényi-entropy approach used by Zhang et al. is unavailable in this setting. However, the polynomial envelope still makes (\int p_t^\alpha) finite for suitable (\alpha<1), while the exponential envelope gives finiteness for every (\alpha>0). Could the authors clarify which step fails and which estimates require new arguments?

Our statement in Appendix A.1 refers specifically to the direct applicability of the sub-Gaussian Rényi-entropy estimates in Zhang et al., rather than the general Rényi-entropy approach. We will refine the wording to make this distinction clearer.

More precisely, the Rényi/Markov and Hölder arguments remain valid under our assumptions. The key limitation arises in the small-order Rényi-entropy estimates used in Zhang et al. (Lemmas F.15–F.17). Writing m=d+γ+1m=d+\gamma+1, our polynomial-tail assumption guarantees ∫ptα<∞\int p_t^\alpha<\infty for α>d/m\alpha>d/m, but this integrability is not guaranteed for α≤d/m\alpha\le d/m. Consequently, the α→0\alpha\to0 argument underlying their near-linear dependence on the density threshold cannot be directly applied. Nevertheless, an adapted Rényi argument with α\alpha approaching d/md/m can recover the polynomial-tail exponent up to logarithmic factors. Similarly, under exponential tails, the Rényi approach remains applicable with appropriately modified entropy estimates.

Our proof instead derives the required bounds directly from the Gaussian-smoothed density envelopes in Lemmas 20 and 23. Specifically, Lemmas 16-17 control the bulk and low-density contributions under polynomial tails, while Lemmas 21-22 establish the corresponding bounds under exponential tails, with explicit dependence on diffusion time. We will revise Appendix A.1 to clarify which estimates require modification and how our analysis differs from the general Rényi-entropy approach.

I thank the authors for the response. It addresses my main questions. I maintain my original positive score.

Comments on the paper

1 comment
Area_Chair_cNZbOpenReview ↗

Discussion phase coming to an end

Dear reviewer Mfto,

As we are approaching the deadline of the discussion phase, please read the authors’ responses and share your thoughts in the discussion as soon as possible. In particular, please clarify whether the responses have addressed your main concerns. Please also update your review and scores if appropriate.

Your timely input will help us reach a well-informed decision. Thank you for your time and effort!

Best regards, Area Chair

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.