- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper investigates the conditioning dependence of diffusion-based sampling compared with classical Langevin algorithms in the Gaussian setting. The authors establish non-asymptotic Wasserstein-2 convergence bounds for the unadjusted Langevin algorithm (ULA), underdamped Langevin dynamics (ULD), and a diffusion-based sampler. Under the proposed hyperparameter choices, ULA and ULD exhibit condition-number dependence, whereas the diffusion sampler achieves a condition-number-free convergence rate. The authors further establish matching asymptotic expressions, demonstrating the sharpness of the derived rates for the specified algorithms and parameter choices. In addition, the paper compares standard and noised score matching under gradient descent, suggesting that the benefit of noising primarily arises during sampling rather than learning.
Critical strengths
-
The paper provides non-asymptotic Wasserstein bounds together with matching asymptotic expressions. In particular, the asymptotic equivalence establishes the sharpness of the diffusion bound, including the leading constant. These results constitute a meaningful theoretical contribution.
-
The analysis highlights how time-dependent scores progressively change the conditioning of the sampling problem, providing a useful perspective on the potential advantages of diffusion-based sampling.
Critical weaknesses
-
The main comparison considers ULA and Euler–Maruyama-discretized ULD with constant step sizes, whereas the diffusion sampler employs a geometrically decreasing step-size schedule. Although these choices are reasonable, the comparison does not establish whether the observed improvement arises specifically from the time-dependent score or partly from the discretization strategy. In particular, it remains unclear whether Langevin algorithms with appropriately chosen nonuniform step sizes could achieve improved condition-number dependence. Moreover, the ULD comparison is restricted to the Euler–Maruyama discretization, while more sophisticated integrators are known to improve the convergence behavior of underdamped Langevin methods. Consequently, the results establish a separation between the specific schemes considered, rather than a general advantage of diffusion over Langevin sampling algorithms.
-
Section 3 shows that, in the Gaussian setting, noised score matching corresponds to ridge-regularized score matching and that some of its regularization effects can be mimicked by early stopping in ordinary score matching. However, this relationship does not establish that noising offers no benefit during learning. Specifically, the two procedures estimate different score functions, and the paper does not compare their finite-sample estimation errors or optimization efficiency. Similarity between the resulting covariance estimators does not necessarily imply comparable statistical performance. Therefore, the conclusion that the benefits of noising arise from sampling rather than learning appears stronger than what Theorem 7 establishes.
Questions for authors
-
Could the authors clarify whether the condition-number separation persists when Langevin algorithms are allowed to use nonuniform step sizes? In particular, to what extent is the improvement attributable to the time-dependent score rather than the choice of discretization schedule?
-
In what precise sense does Theorem 7 establish that noising provides no benefit during learning? Can the authors clarify whether the similarity between the resulting covariance estimators implies comparable statistical estimation errors, given that the two methods estimate different score functions?
2 comments
Thank you for your response. I maintain my positive assessment and score.
We thank the reviewer for the positive assessment of our work and insightful questions.
Q1: We agree that the analysis for other discretization schemes and step size schedules for Langevin is important. We focused on the Euler-Maruyama with uniform step size for simplicity. The main limitation for a more general analysis is that the eigenvalues of the covariance matrices cannot be computed in closed form.
Regarding the discretization schemes, as discussed in our related work section, the literature on Langevin has provided improved condition dependence for Gaussian targets with more complex integrators (splitting, random step-sizes, or Metropolis corrections). To our knowledge, these methods do not yield independence from the condition number, and are arguably more complex than diffusions.
Regarding the step size schedule, we believe that nonuniform step sizes for Langevin do not significantly improve the convergence rates and do not close the gap with diffusions in terms of dependence on the condition number. More specifically, one can show that ULA suffers a κlog(κ) burn-in for any step size schedule as long as the steps remain stable. Furthermore, we investigated convergence rates for ULA with nonuniform step size ηk=c/k, where c is a tunable hyperparameter. In this setting, the computations remain simple enough. We find that, depending on the value of c compared with λmax/2, there are two regimes. In the first one, we have W2(PN,P)∼A(c)/N, with A(c)≥81λmaxTr(Σ−1), which is the same prefactor as ULA with uniform step sizes. In the second one, we obtain W2(PN,P)∼B(c)N−2c/λmax. Choosing c such that 2c/λmax is close to 1 causes the quantity B(c) to explode.
We will add this discussion to the final version, and state that a lower bound for Langevin-type methods to prove a general separation with diffusions is an interesting next step.
Q2: We have rephrased the sentence mentioned by the reviewer (in the abstract and in the contributions) to clarify that we study the impact of the noise through the lens of the implicit bias of the learning objective, and show that noising does not add an implicit bias. In a data-limited or compute-limited scenario, we agree that one could also study the effect of noising on the estimation error or optimization efficiency. These are interesting avenues for future work that we will mention in the final version.