- Rating
- 3(+1)
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
The paper studies score estimation via a thresholded Gaussian KDE, together with total-variation sampling guarantees for the continuous-time reverse SDE, for Sobolev-smooth targets with exponential or polynomial tail decay. Minimax lower bounds for score estimation are also derived in both regimes, matching the upper bounds up to logarithmic factors over restricted ranges of diffusion time.
Critical strengths
- The problem is well-motivated. Nearly all existing minimax theory for diffusion models assumes sub-Gaussian tails, compact support, or a density lower bound. This paper is a clean first step toward heavy tails within the standard Gaussian pipeline.
- The score-estimation results are sharp in a meaningful sense. Upper and lower bounds match up to polylog factors in both regimes. The lower-bound construction for polynomial tails is the most original technical part.
Critical weaknesses
- The polynomial-tail sampling rate seems to be strictly suboptimal, not just of unknown optimality. Any sampler's output law is a density estimator, so classical density estimation gives a benchmark. A second-order KDE with the standard truncation argument works as follows. Bound the error on a ball of radius by , add the tail mass , and balance the two. We have . That exponent's denominator, , is smaller than Theorem 4's, , so the naive KDE rate is strictly faster for every . This matters for the paper's framing. It suggests the diffusion pipeline, or at least this analysis of it, loses something under heavy tails. Comparing against the KDE baseline would sharpen the discussion.
Questions for authors
- There is no discretization analysis in this work. The continuous-time reverse process is usually discretized to perform generative sampling. Would the authors provide the discretization error analysis or explain why such an analysis is not necessary?
- The forward process seems too simple. In practice, people would choose non-constant or even time-dependent drift and diffusion coefficients. Do the theoretical results extend to such general settings? If so, could the authors discuss the required assumptions and any additional technical challenges?
4 comments
Reply to Reviewer roTM Part 2
-
The forward process $dX_t=dB_t$ seems too simple. In practice, people would choose non-constant or even time-dependent drift and diffusion coefficients. Do the theoretical results extend to such general settings? If so, could the authors discuss the required assumptions and any additional technical challenges?
Our choice of Brownian motion is deliberate, as our primary objective is to characterize the statistical effects of heavy-tailed targets on score estimation and sampling under Gaussian noising, rather than to investigate different forward dynamics.
Importantly, the Gaussian-noising structure underlying our analysis also applies to a broader class of forward processes, including commonly used variance-exploding and variance-preserving diffusions. More generally, consider the linear SDE , where and are deterministic scalar functions. Define
\qquad v_t=\int_0^t\frac{b(s)^2}{\alpha_s^2}ds.$$ Assume $a$ is locally integrable, $b/\alpha$ is locally square-integrable, and $v_t$ is strictly increasing. Then, for each $t$, $$X_t\overset{d}{=}\alpha_t(X_0+\sqrt{v_t}Z), \qquad Z\sim N(0,I_d),\quad Z\perp X_0.$$ Let $q_v=p_0*\varphi_v$, where $\varphi_v$ denotes the density of $\mathcal{N}(0,vI_d)$, and define $s_v=\nabla\log q_v$. Then, the density $p_t^X$ of $X_t$ and its associated score $s_t^X$ are given by $$p_t^X(x)=\alpha_t^{-d}q_{v_t}(x/\alpha_t),\quad s_t^X(x)=\alpha_t^{-1}s_{v_t}(x/\alpha_t).$$ Consequently, the transformed estimator $\hat s_t^X(x)=\alpha_t^{-1}\hat s_{v_t}(x/\alpha_t)$ has weighted squared score risk scaled by $\alpha_t^{-2}$. Our upper and minimax lower bounds therefore transfer with $t$ replaced by $v_t$, provided the transformed times and distribution classes satisfy the respective theorem assumptions. For sampling, the diffusion-weighted cumulative score error satisfies $$\int_{t_0}^{T}b(t)^2\lVert \hat s_t^X-s_t^X \rVert_{L^2(p_t^X)}^2\,dt =\int_{v_{t_0}}^{v_T} \lVert \hat s_v-s_v \rVert_{L^2(q_v)}^2\,dv.$$ This is precisely the score-error term appearing in the corresponding Girsanov bound. Extending the sampling guarantees further requires appropriate transformations of the reverse dynamics and initialization, as well as rescaling the stopped output for comparison with $p_0$. These adjustments must be taken into account when establishing sampling rates under general noise schedules. In contrast, more general SDEs with nonlinear drifts or state-dependent diffusion coefficients need not preserve the Gaussian-convolution structure. Extending our analysis to such settings would require additional estimates for the transition densities, their tail behavior, and the associated score functions, together with appropriate modifications to the reverse-time sampling analysis. We will clarify these distinctions in the revised manuscript.My concerns are properly addressed. I raised the score.
Thank you for your positive feedback and for reconsidering your assessment.
Reply to Reviewer roTM Part 1
We thank the reviewer for the feedback. Please find our point by point response below.
We agree that the proposed direct KDE-based distribution estimator achieves a faster TV convergence rate than the upper bound in Theorem 4, showing that our bound is not minimax-sharp as a general distribution estimation guarantee. However, comparing two upper bounds does not establish which sampling procedure actually converges faster. In particular, although the direct KDE estimator admits a stronger theoretical guarantee, Theorem 4 only provides an upper bound for our reverse-diffusion sampler driven by a thresholded KDE score, rather than a matching characterization of its actual convergence rate. Therefore, this comparison alone does not establish that our proposed diffusion sampler is statistically less efficient.
Moreover, the same KDE rate can be achieved using the empirical Gaussian KDE p^t(x)=n1∑i=1nφt(x−Xi), which admits an exact reverse-diffusion representation through its unthresholded score when initialized from the corresponding smoothed empirical distribution. Thus, the faster KDE rate does not imply an intrinsic limitation of Gaussian diffusion under heavy tails. Rather, the gap may arise from our score-to-TV analysis, the particular thresholded-score construction, or both.
As outlined in the Introduction, our primary objective is to characterize the minimax-optimal rates for score estimation under heavy-tailed distributions and investigate how these estimation errors propagate to sampling accuracy. Our thresholded score estimator is designed to control errors in low-density regions and achieves the minimax-optimal score-estimation rate established in Theorems 1 and 9. Theorem 4 then derives a TV sampling guarantee through Girsanov's theorem and Pinsker's inequality, which may not yield a sharp bound for the proposed sampler. We do not claim minimax optimality of this sampling guarantee, as acknowledged in the abstract and Discussion.
We appreciate the suggested KDE comparison and will incorporate it into the revised manuscript.
Our primary objective is to characterize the fundamental statistical limits of score estimation under heavy-tailed distributions and investigate how these estimation errors propagate to sampling accuracy. We therefore focus on the continuous-time reverse SDE, which allows us to isolate the statistical error from numerical discretization errors.
Discretization introduces a separate computational question beyond the statistical objectives of this work. While important for practical implementations, its analysis is not required to establish our minimax score-estimation rates or continuous-time sampling guarantees. As acknowledged in the Discussion, extending our results to discretized samplers is an interesting direction for future work.