acceptodds
Under review as a conference paper at ICLR 2027

Denoising Levy Scores for Generative Modeling

11 views

Abstract

We investigate Levy-driven sampling to improve sample quality in score-based generative modeling under a fixed function-evaluation budget. Levy jumps enable long-range transitions between separated regions, offering an alternative to Brownian Langevin relaxation at each noise level. Unlike existing Levy-based generative models, which introduce jumps through non-Gaussian forward noising, we retain Gaussian perturbations and introduce Levy Langevin correctors (LLCs) only at sampling step. These correctors depend only on the Levy score, a nonlocal analogue of the Stein score. We develop denoising Levy score matching (DLSM) by perturbing the data with different levels of Gaussian noise and jointly estimating the corresponding Levy scores. On synthetic multimodal and heavy-tailed target distributions, our method achieves up to an reduction in error relative to Brownian samplers at matched NFEs. At matched low-NFE budgets, our method achieves lower FID than Brownian samplers on CIFAR-10, CelebA-64, and CIFAR-10-LT; on CIFAR-10-LT, it also reproduces the long-tailed class distribution more accurately.

Primary area:
generative models
Keywords:
score-based generative modeling, diffusion models, denoising score matching, Levy processes, jump processes, nonlocal sampling, Langevin dynamics

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_9Uw4OpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

The paper proposes GLSM, which retains the standard Gaussian forward diffusion and introduces Lévy jumps with a prescribed measure ν\nu only in the generative dynamics. Building on the Lévy score of Huang et al. and the identity Lν∗p=∇⋅(p Sν[p])\mathcal{L} _{\nu}^* p =\nabla\cdot(p\,S _\nu[p]), it derives:

  • a marginal-preserving reverse-time Lévy–Itô SDE (Thm 3.1);
  • a denoising objective (DLSM) that learns Sν[pt]S_\nu[p_t] from standard DSM pairs, with closed-form targets for compound-Poisson and α\alpha-stable jumps (Thm 3.2);
  • an annealed pure-jump Lévy Langevin sampler (Thm 3.3, Alg. 1).

Experiments cover two 2D targets, FID versus NFE on CIFAR-10, CelebA-64 and CIFAR-10-LT, and long-tail metrics on CIFAR-10-LT. All image experiments use compound-Poisson jumps with Gaussian amplitudes.

Critical strengths

S1. The core idea is simple and practical. The Gaussian prior, the DSM data pipeline and standard architectures are reused unchanged; only the regression target and the sampling update differ.

S2. The learning objective is technically well executed. The closed-form conditional targets for compound-Poisson and α-stable jumps, and the Bregman-divergence treatment of the infinite-variance stable target (Appendix B.3), are clean and reusable.

S3. The 2D experiments (Fig. 2, Table 1) show clear gains in heavy-tail and low-mass-mode recovery: sliced W1 is reduced by 9.4× and tail MSLE by roughly 38× on the Cauchy target.

S4. On images, GLSM consistently outperforms the stochastic baselines in the 128–256 NFE range on all three datasets, and the ablations over ladder resolution, updates per level and σmax\sigma_max are reasonably extensive.

Critical weaknesses

W1. (Gap between the headline theorem and the algorithm actually used.) The experimental sampler (Alg. 1) does not use the jump-diffusion reverse SDE of Thm 3.1; it only anneals the pure-jump corrector. Moreover, Figs. 6–7 and Sec. D.3.3 show that GLSM works only in the regime of a fine ladder with a single update per level (L≥16L \ge 16, M=1M = 1), whereas few levels with many updates fail for both methods (FID 200–500 for L≤8L \le 8). Hence the interpretation of the LLC as a corrector that relaxes toward ptp_t (Thm 3.3) does not describe how the method operates in practice. In this regime the sampler effectively acts as a reverse-time integrator, yet the theorem relevant to that view is never used.

Questions for authors

Q1. Could the authors report FID for the PF-ODE (Heun/RK45), DDIM and DPM-Solver++ on the same NCSN++ checkpoint at 32–256 NFE? Fig. 4 compares only stochastic samplers (Table 9 already includes DDIM and DPM-Solver(++) for coverage), so the fixed-budget quality claim cannot be assessed.

Q2. The compound-Poisson increment on images (Eqs. 21–22) is a Poisson scale mixture of Gaussians with the same mean covariance 2h2h as SMLD's noise, and the Eq. (18) target is a radially reweighted score −c(∥ε∥) ε-c(\|\varepsilon\|)\,\varepsilon. Could the authors isolate the source of the gain at fixed (L,M)(L, M), e.g. (128,1)(128, 1), via (a) Lévy-score drift with Gaussian noise, (b) Stein-score drift with compound-Poisson noise, and (c) removing the soft clipping?

4 comments
AuthorsOpenReview ↗

Response to Reviewer 9Uw4 — Part 1/2

We thank the reviewer for recognizing the theoretical and practical contributions of GLSM. We address the concerns below.

(W1) Theorem 3.1, Algorithm 1, and the annealing schedule. We would like to clarify that Theorem 3.1 and Algorithm 1 describe two distinct sampling dynamics, rather than a continuous-time SDE and its numerical discretization. This is directly analogous to the distinction between reverse-time SDE sampling [1] and annealed Langevin sampling [2] in conventional Brownian score-based models.

Theorem 3.1 constructs a reverse Lévy SDE whose time marginals exactly match the Gaussian marginal path pT−sp_{T-s}: dYs=[−f(Ys−,T−s)+g2(T−s)+h2(s)2S[pT−s](Ys−)+SνT−s[pT−s](Ys−)]ds+h(s)dWs+dJs.dY_s=\bigg[-f(Y_{s-},T-s)+\frac{g^2(T-s)+h^2(s)}{2}S[p_{T-s}]{}(Y_{s-})+S_{\nu_{T-s}}[p_{T-s}]{}(Y_{s-})\bigg]ds+h(s)dW_s+dJ_s.

Writing ti=T−sit_i=T-s_i, its Euler discretization is Ysi+1=Ysi+Δsi[−f(Ysi,ti)+g2(ti)+h2(si)2S[pti](Ysi)+Sνti[pti](Ysi)]+h(si)Δsi ξi+ΔJi.Y_{s_{i+1}}=Y_{s_i}+\Delta s_i\bigg[-f(Y_{s_i},t_i)+\frac{g^2(t_i)+h^2(s_i)}{2}S[p_{t_i}]{}(Y_{s_i})+S_{\nu_{t_i}}[p_{t_i}]{}(Y_{s_i})\bigg]+h(s_i)\sqrt{\Delta s_i}\mkern3mu\xi_i+\Delta J_i.

Algorithm 1 (LLC) instead follows the annealed Langevin sampling paradigm. At each noise level tit_i, we consider the continuous-time LLC targeting the fixed distribution ptip_{t_i}: dYτ(i)=Sνti[pti](Yτ−(i)) dτ+dLτνti.dY_\tau^{(i)}=S_{\nu_{t_i}}[p_{t_i}]{}(Y_{\tau-}^{(i)})\mkern3mu d\tau+dL_\tau^{\nu_{t_i}}.

Its Euler updates are Yi,m+1=Yi,m+ηiSνti[pti](Yi,m)+ΔLi,mνti,Yi+1,0=Yi,Mi.Y_{i,m+1}=Y_{i,m}+\eta_iS_{\nu_{t_i}}[p_{t_i}]{}(Y_{i,m})+\Delta L_{i,m}^{\nu_{t_i}},\qquad Y_{i+1,0}=Y_{i,M_i}.

Our image experiments use Mi=1M_i=1, giving Yi+1=Yi+ηiSνti[pti](Yi)+ΔLi.Y_{i+1}=Y_i+\eta_iS_{\nu_{t_i}}[p_{t_i}]{}(Y_i)+\Delta L_i.

Thus, even when Mi=1M_i=1, LLC remains an annealed Langevin sampler, not a discretization of the reverse SDE. In particular, the LLC update does not contain the original forward drift or the Stein-score term required by Theorem 3.1. We have clarified this distinction in Section 3.1 (lines 247-253) and Section 3.3 (lines 308-339).

The annealing procedure can be understood as tracking a sequence of gradually changing target distributions. Several works have established non-asymptotic guarantees for Brownian annealed Langevin sampling [3-5]. These results show that accurate final sampling can be achieved without full equilibration at every intermediate target under suitable conditions.

We also conducted empirical ablation studies of the annealing parameters LL and MM (Appendix D.3.1-D.3.3, Table 7 and Figures 6-7). Fixing L=32L=32 and increasing MM from 1 to 32 improves CIFAR-10 validation FID from 92.792.7 to 20.620.6, compared with 97.197.1 for SMLD at M=32M=32. Even with only L=16L=16 levels and M=128M=128 updates per level, GLSM achieves an FID of 23.923.9, compared with 89.989.9 for SMLD.

When fixing the total budget LMLM at 128 or 256 NFEs, allocating more evaluations to distinct noise levels yields better results, with M=1M=1 achieving the lowest FID among the tested configurations (Figures 6-7). Therefore, M=1M=1 is a numerical choice motivated by efficiency under a fixed computational budget, rather than a requirement of the LLC construction.

[1] Song, Y., et al. (2021). Score-Based Generative Modeling through Stochastic Differential Equations. ICLR.

[2] Song, Y., and Ermon, S. (2019). Generative Modeling by Estimating Gradients of the Data Distribution. NeurIPS.

[3] Guo et al., Provable Benefit of Annealed Langevin Monte Carlo for Non-Log-Concave Sampling, ICLR 2025.

[4] Cordero-Encinar et al., Non-asymptotic Analysis of Diffusion Annealed Langevin Monte Carlo for Generative Modelling, 2025.

[5] Nitanda et al., Slowly Annealed Langevin Dynamics: Theory and Applications to Training-Free Guided Generation, NeurIPS 2026.

AuthorsOpenReview ↗

Response to Reviewer 9Uw4 — Part 2/2

(Q1) Comparison with deterministic diffusion solvers. We thank the reviewer for suggesting stronger deterministic baselines. We have evaluated DDIM, Heun PF-ODE, and DPM-Solver++ using the same frozen NCSN++ VE checkpoint as our main CIFAR-10 experiment. The results are summarized below.

FID ↓\downarrow3264128256
DDIM / Euler PF-ODE10.348.838.298.11
DPM-Solver++ (2M)8.298.047.967.97
Heun PF-ODE8.147.977.937.94
GLSM (ours)81.834.110.076.47

These deterministic solvers are specifically designed to accelerate probability-flow ODE sampling through higher-order numerical integration. Their superior performance at very low NFEs is therefore expected. Nevertheless, at 256 NFEs, GLSM achieves an FID of 6.476.47, outperforming Heun (7.947.94) and DPM-Solver++ (7.977.97).

Our main contribution for this paper is a new stochastic generative framework based on Lévy dynamics, rather than a high-order numerical solver, thus our primary comparisons focus on stochastic samplers. Developing more efficient numerical integration schemes for Lévy dynamics, including both the marginal-preserving reverse SDE in Theorem 3.1 and the annealed LLC in Algorithm 1, is a separate research direction that we are currently pursuing.

(Q2) Ablation of Lévy scores, jump noise, and clipping. Theoretically, to preserve the desired invariant distribution ptp_t, the drift and noise generator must be matched to satisfy the stationary Fokker-Planck equation: −∇⋅(ptS[pt])+Δpt=0,−∇⋅(ptSνt[pt])+Lνt∗pt=0.-\nabla\cdot(p_tS[p_t])+\Delta p_t=0,\qquad -\nabla\cdot(p_tS_{\nu_t}[p_t])+\mathcal L_{\nu_t}^{\ast}p_t=0. Using a mismatched drift-noise pair generally fails to preserve the target distribution ptp_t. Numerically, we apply soft clipping to stabilize the drift updates, consistent with the use of clipping in Lévy-based generative models such as LIM/DLPM [6,7].

To further distinguish the contributions of the Lévy score and jump noise, we performed the requested ablations on CIFAR-10 at NFEs ranging from 128 to 2048, using the same trained scores and sampling hyperparameters.

Configuration12825651210242048
Lévy drift + CP noise (ours)10.076.475.074.534.44
Lévy drift + Gaussian noise (a)9.475.245.274.764.55
Stein drift + CP noise (b)328.4309.1161.3139.0127.7
GLSM without soft clipping (c)370.2365.2330.47.184.52

The results show that (b) performs substantially worse, while (c) confirms the importance of soft clipping for numerical stability at low NFEs. Interestingly, (a) achieves slightly better FID at 128/256 NFEs, but GLSM outperforms it at 512–2048 NFEs. The similar performance at low NFEs may be attributed to two factors: (1) errors introduced by score learning and numerical clipping; and (2) our compound-Poisson increments are Poisson mixtures of Gaussians with the same covariance 2hId2hI_d as Brownian increments, making similar finite-step performance plausible under our sampling parameters.

However, the finite-step FID comparison alone cannot establish whether (a) preserves the correct target distribution. To further verify the importance of matching the drift and noise, we conducted an experiment on an analytically tractable 2D Gaussian mixture using exact Stein and Lévy scores. Starting from pσp_\sigma, we fix the corrector time Tc=σ2T_c=\sigma^2 and refine the timestep. At σ=0.25\sigma=0.25, we obtain:

GLSM (ours)(a)(b)
Sliced W1↓W_1\downarrow0.00510.03470.0554

(a) and (b) retain substantially larger errors under timestep refinement, which confirms that they fail to preserve the target distribution even with exact scores.

Moreover, the computational overhead of compound-Poisson noise is negligible. On an NVIDIA H200 GPU with batch size 1024 at 128 NFEs, GLSM takes 20.87 ms per image, compared with 20.72 ms for (a), an overhead of only about 0.7%. Thus, there is little computational incentive to sacrifice the correct invariant distribution by replacing the matched Lévy noise with Gaussian noise.

[6] Yoon et al., Score-based Generative Models with Lévy Processes, NeurIPS 2023.

[7] Shariatian et al., Denoising Lévy Probabilistic Models, ICLR 2025.

Thank you for the detailed responses and additional experiment results. The clarification between Theorem 3.1 and Algorithm 1 make me deeply understand the paper as well. Most of my concerns are resolved in this discussion period. Can the authors add the deterministic-solver comparison from the rebuttal to the paper and focus the overall contribution of the paper on stochastic reverse process?

We thank the reviewer for the encouraging feedback and constructive suggestions. We have added the deterministic-solver comparisons (DDIM, Heun PF-ODE, and DPM-Solver++) to Appendix D.3 of the revised manuscript.

We also agree that the paper should be positioned primarily as a framework for stochastic generative dynamics, rather than as an attempt to develop faster numerical solvers for existing probability-flow ODEs. Our main contribution is to incorporate Lévy jumps into generation while retaining the standard Gaussian forward process. Specifically, Theorem 3.1 establishes marginal-preserving Lévy jump-diffusion reverse dynamics, while Algorithm 1 provides a distinct stochastic sampling construction based on annealed Lévy Langevin correctors. We have further clarified this distinction and emphasized the stochastic perspective in the revised manuscript.

We are glad that our responses have addressed most of your concerns. We hope these additional revisions also address your remaining suggestions, and would be grateful if you would consider revisiting your rating.

Reviewer_WufsOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes the Gaussian-Lévy Score Model (GLSM): it retains the standard Gaussian forward diffusion and its tractable transition kernel, DSM training objective, and terminal Gaussian reference distribution, but introduces a Lévy-Ito SDE with an arbitrary admissible Lévy measure and a Lévy Langevin corrector (LLC) in the reverse generation stage. The core is to define a nonlocal Lévy score and prove that, for any given Lévy measure, one can construct a reverse jump-diffusion whose time marginals exactly match those of the Gaussian forward diffusion (Theorem 3.1). The authors further provide the denoising Lévy score identity (Theorem 3.2), showing that the conditional Lévy score can be regressed from exactly the same Gaussian clean-noisy pairs as in standard DSM, yielding DLSM; they also prove that the pure-jump LLC with a fixed target p_t has p_t as its unique invariant distribution (Theorem 3.3). During sampling, the pure-jump LLC is annealed over decreasing diffusion times. Experiments cover Cauchy heavy-tailed targets, non-equilibrium Gaussian mixtures, low-NFE generation on CIFAR-10/CelebA-64, and CIFAR-10-LT long-tail coverage and class-conditional FID, showing improvements over baselines such as SMLD, NCSN++, DDPM, LIM, and DLPM under matched NFE budgets.

Critical strengths

The construction is elegant and the theoretical contribution is clear. Retaining Gaussian forward diffusion while constructing a reverse jump-diffusion with an arbitrary admissible Lévy measure that exactly matches the Gaussian marginals conceptually decouples the “forward perturbation law” from the “generative dynamics.” Theorem 3.1 provides exact marginal matching, Theorem 3.2 provides a trainable conditional regression objective, and Theorem 3.3 provides LLC invariance; together they form a relatively complete “construction–learning–sampling” chain.

The experiments are relatively broad, with substantial gains in low-NFE and long-tail settings. In the low-NFE regime on CIFAR-10/CelebA-64, the FID-NFE curves of GLSM are clearly better than those of the compared methods; on CIFAR-10-LT, coverage is 0.868, Tail-3 coverage is 0.967, and average class FID is 10.8, which are large improvements over DLPM/LIM/NCSN++. These results support the claim that “the jump law can serve as an additional design dimension.”

Critical weaknesses

There is an unquantified finite-step bias between the exact reverse dynamics and the actual annealed sampler. The exact reverse SDE in Theorem 3.1 includes Stein score, Lévy score, and Brownian/jump terms, whereas Algorithm 1 uses a pure-jump LLC with a fixed target p_t and switches to the next noise level after a finite number of steps. Theorem 3.3 guarantees convergence only with infinitely many steps per stage; the finite-step error of the annealed warm start, the step size n_{L_i}, the number of updates per stage M_{L_i}, and the degree to which the final sample deviates from p_{L_i}(t_{L_i,min}) are not theoretically or systematically empirically calibrated.

Experimental fairness and the choice of jump law need more sufficient explanation. GLSM requires choosing a jump measure and its time schedule (e.g., a_0 for compound-Poisson and α for α-stable), while baselines are also tuned under their own native parameterizations; the main text does not sufficiently show the per-step jump sampling overhead, wall-clock time, or whether the advantage persists in the high-NFE regime. The class-conditional FID on CIFAR-10-LT uses a frozen classifier to assign generated sample classes and compares them with balanced real classes; this protocol may amplify the advantage of methods with balanced coverage, and it should be clarified whether it is applied consistently to all methods.

Questions for authors

(1) Is there a quantifiable finite-step bias between Algorithm 1 and the exact reverse SDE in Theorem 3.1? Can you provide a finite-step error bound for the LLC at each stage, or at least systematically compare the deviation of the final sample from p_t(min) under different M, n, and numbers of annealing levels L? (2) How is ν_t scheduled over diffusion time t in the experiments? Are the compound-Poisson parameters λ_t, ρ_t, τ_t} and the α-stable parameter α fixed or scaled with σ_t? Does the choice of jump law on different targets rely on validation-set tuning? Can you provide a selection criterion? (3) Beyond matching the number of network evaluations (NFE), how does the wall-clock time of GLSM’s per-step jump sampling compare with baselines? Does GLSM still maintain an advantage in the high-NFE regime (e.g., 512/1024), or does it only outperform at low budgets? (4) The class-conditional FID on CIFAR-10-LT uses a frozen classifier to assign generated classes; is this protocol exactly the same for GLSM and the other methods? Is it possible that classifier bias amplifies the advantage of methods with balanced coverage? Can you add classifier-independent coverage/recall or human evaluation? (5) The paper claims “arbitrary admissible Lévy measure,” but the experiments only cover compound-Poisson Gaussian jump amplitudes and isotropic α-stable. Can you add qualitative/quantitative results for at least one other Lévy measure (e.g., Gamma or a mixed jump law) to support the operational meaning of “arbitrary”? (6) The core difference from LIM/DLPM is “Gaussian forward, Lévy reverse.” Can you perform a stricter ablation under the same training budget, same backbone, and same jump law to separate the contribution of “Lévy generative dynamics” from “additional tuning/architecture”?

3 comments
AuthorsOpenReview ↗

Response to Reviewer Wufs — Part 1/3

We thank the reviewer for recognizing the theoretical contributions of GLSM and its improvements in low-NFE generation and long-tail coverage. We address the concerns below.

(Q1) Theorem 3.1, Algorithm 1, and finite-step convergence. Theorem 3.1 and Algorithm 1 (LLC) correspond to two distinct sampling schemes, analogous to reverse-time SDE sampling (Song et al. [1]) and annealed Langevin sampling (Song and Ermon [2]) in conventional Brownian score-based models.

Theorem 3.1 constructs a reverse Lévy SDE whose time marginals exactly match the Gaussian marginal path pT−sp_{T-s}: dYs=[−f(Ys−,T−s)+g2(T−s)+h2(s)2S[pT−s](Ys−)+SνT−s[pT−s](Ys−)]ds+h(s)dWs+dJs.dY_s=\bigg[-f(Y_{s-},T-s)+\frac{g^2(T-s)+h^2(s)}{2}S[p_{T-s}]{}(Y_{s-})+S_{\nu_{T-s}}[p_{T-s}]{}(Y_{s-})\bigg]ds+h(s)dW_s+dJ_s.

Writing ti=T−sit_i=T-s_i, its Euler discretization is Ysi+1=Ysi+Δsi[−f(Ysi,ti)+g2(ti)+h2(si)2S[pti](Ysi)+Sνti[pti](Ysi)]+h(si)Δsi ξi+ΔJi.Y_{s_{i+1}}=Y_{s_i}+\Delta s_i\bigg[-f(Y_{s_i},t_i)+\frac{g^2(t_i)+h^2(s_i)}{2}S[p_{t_i}]{}(Y_{s_i})+S_{\nu_{t_i}}[p_{t_i}]{}(Y_{s_i})\bigg]+h(s_i)\sqrt{\Delta s_i}\mkern3mu\xi_i+\Delta J_i.

Algorithm 1 (LLC) instead follows the annealed Langevin sampling paradigm. At each noise level tit_i, we consider the continuous-time LLC targeting the fixed distribution ptip_{t_i}: dYτ(i)=Sνti[pti](Yτ−(i)) dτ+dLτνti.dY_\tau^{(i)}=S_{\nu_{t_i}}[p_{t_i}]{}(Y_{\tau-}^{(i)})\mkern3mu d\tau+dL_\tau^{\nu_{t_i}}.

Its Euler updates are Yi,m+1=Yi,m+ηiSνti[pti](Yi,m)+ΔLi,mνti,Yi+1,0=Yi,Mi.Y_{i,m+1}=Y_{i,m}+\eta_iS_{\nu_{t_i}}[p_{t_i}]{}(Y_{i,m})+\Delta L_{i,m}^{\nu_{t_i}},\qquad Y_{i+1,0}=Y_{i,M_i}.

Our image experiments use Mi=1M_i=1, giving Yi+1=Yi+ηiSνti[pti](Yi)+ΔLi.Y_{i+1}=Y_i+\eta_iS_{\nu_{t_i}}[p_{t_i}]{}(Y_i)+\Delta L_i.

Thus, LLC is not a discretization of the reverse SDE. We have clarified this distinction in Section 3.1 (lines 247-253) and Section 3.3 (lines 308-339). In generation stage, the reverse SDE requires both Stein-score and Lévy-score networks (two NFEs per step), whereas LLC requires only the Lévy-score network (one NFE per step). We therefore adopt LLC for efficient generation.

Regarding finite-step convergence, several works have established non-asymptotic guarantees for Brownian annealed Langevin sampling [3-5]. These results show that accurate final sampling can be achieved without full equilibration at every intermediate target under suitable conditions.

For our LLC, a similar analysis can be formulated. Let qi=ptiq_i=p_{t_i} be the target at level ii, μi\mu_i the sampling distribution after that level, and ei=D(μi,qi)e_i=D(\mu_i,q_i) for a suitable distribution metric DD. Assuming the exact LLC contracts toward qiq_i at rate κi\kappa_i, with controlled discretization and score-estimation errors, a natural error recursion is ei≤ρiei−1+ρiD(qi−1,qi)+εidisc+εiscore,ρi=e−κiMiηi.e_i\leq \rho_i e_{i-1}+\rho_i D(q_{i-1},q_i)+\varepsilon_i^{\mathrm{disc}}+\varepsilon_i^{\mathrm{score}},\qquad \rho_i=e^{-\kappa_iM_i\eta_i}.

Iterating this inequality would separate the final sampling error into four contributions: the initial mismatch between the Gaussian prior and pTp_T, the tracking error caused by changes between successive targets, the discretization error from finite numerical updates, and the error from the learned Lévy score. The contraction factors ρi\rho_i determine how these errors accumulate and decay throughout annealing. Establishing this bound requires quantitative contraction and error estimates for the nonlocal Lévy generator. In particular, the dependence of κi\kappa_i, εidisc\varepsilon_i^{\mathrm{disc}}, and εiscore\varepsilon_i^{\mathrm{score}} on the jump law νti\nu_{t_i}, together with their dependence on LL, MiM_i, and ηi\eta_i, must be analyzed specifically for Lévy dynamics. This is an important direction for future work.

We also conducted empirical ablation studies of the annealing parameters LL and MM (Appendix D.3.2-D.3.3, Table 7 and Figures 6-7). First, fixing L=32L=32 and increasing MM from 1 to 32 improves CIFAR-10 validation FID from 92.7 to 20.6. Second, fixing the total budget LMLM at 128 or 256 NFEs and varying its allocation shows that M=1M=1 achieves the lowest FID among the tested configurations on both CIFAR-10 and CIFAR-10-LT. These results indicate that additional updates improve sampling on coarse noise ladders, but allocating a fixed budget to more distinct noise levels is more effective. This supports our choice of M=1M=1 and is consistent with the observation in [3] that full equilibration at intermediate targets is not required for accurate final sampling.

[1] Song, Y., et al. (2021). Score-Based Generative Modeling through Stochastic Differential Equations. ICLR.

[2] Song, Y., and Ermon, S. (2019). Generative Modeling by Estimating Gradients of the Data Distribution. NeurIPS.

[3] Guo et al., Provable Benefit of Annealed Langevin Monte Carlo for Non-Log-Concave Sampling, ICLR 2025.

[4] Cordero-Encinar et al., Non-asymptotic Analysis of Diffusion Annealed Langevin Monte Carlo for Generative Modelling, 2025.

[5] Nitanda et al., Slowly Annealed Langevin Dynamics: Theory and Applications to Training-Free Guided Generation, NeurIPS 2026.

AuthorsOpenReview ↗

Response to Reviewer Wufs — Part 2/3

(Q2) Parameter scheduling and selection of Lévy measures. In this paper, we aim to study the effect of nonlocal jumps rather than simply increasing the noise variance. We therefore match the covariance of compound-Poisson increments to that of Brownian Langevin noise: Cov⁡(ΔJt)=2hId=Cov⁡(2h ξ),ξ∼N(0,Id).\operatorname{Cov}(\Delta J_t)=2hI_d=\operatorname{Cov}(\sqrt{2h}\mkern3mu\xi),\qquad \xi\sim\mathcal N(0,I_d). Specifically, we choose νt(dr)=λtρt(dr),ρt=N(0,τt2Id),τt2=a02σt2d,λt=2τt2.\nu_t(dr)=\lambda_t\rho_t(dr),\qquad \rho_t=\mathcal N(0,\tau_t^2I_d),\qquad \tau_t^2=\frac{a_0^2\sigma_t^2}{d},\qquad \lambda_t=\frac{2}{\tau_t^2}. Thus, the jump amplitude scales linearly with σt\sigma_t, while the jump intensity scales as σt−2\sigma_t^{-2}. For α\alpha-stable noise, α\alpha remains fixed, and the sampling increment scale ht1/α=ησth_t^{1/\alpha}=\eta\sigma_t decreases with σt\sigma_t.

For synthetic experiments, we select a0∈{4,6,8}a_0\in\lbrace 4,6,8\rbrace and α∈{1.5,1.7,1.8,1.9}\alpha\in\lbrace 1.5,1.7,1.8,1.9\rbrace by minimizing the average validation log-sliced-W1W_1 over NFE budgets {16,32,64,128,256}\lbrace 16,32,64,128,256\rbrace. The selected parameters are fixed across budgets, while the step size η\eta is tuned separately for each method and budget (Appendix D.1.2). For image generation, we fix a0=8a_0=8 and tune η\eta using validation FID (Appendix D.2.2).

Our framework accommodates any admissible Lévy noise, here we just use the most simple, standard choices (like Gaussian compound-Poisson, isotropic α\alpha-stable), and obtain substantial improvements over baselines. How to choose the best jump law also depends on the problem, and developing a principled criterion for selecting the optimal Lévy measure is an important direction for future work.

(Q3) Computational cost and high-NFE performance. GLSM uses the same NCSN++ network architecture as the Brownian baseline, and requires one score-network evaluation per sampling step. The only additional sampling cost in GLSM comes from generating compound-Poisson noise. For Gaussian jump amplitudes, the jump increment can be sampled as ΔJi=dNiτti2 Zi,Ni∼Poisson⁡(λtihi),Zi∼N(0,Id).\Delta J_i\overset{d}{=}\sqrt{N_i\tau_{t_i}^2}\mkern3mu Z_i,\qquad N_i\sim\operatorname{Poisson}(\lambda_{t_i}h_i),\quad Z_i\sim\mathcal N(0,I_d). This requires only one Poisson draw and one Gaussian vector, with O(d)O(d) operations, which is inexpensive compared with a network evaluation. We further verified this through wall-clock measurements on an NVIDIA H200 GPU with batch size 1024 at 128 NFEs. GLSM takes 20.87 ms per image, compared with 20.33 ms for SMLD, an overhead of only 2.7%. Thus, NFE-based comparisons closely reflect wall-clock sampling efficiency in our experiments.

Figure 4 was originally intended to highlight that GLSM achieves sample quality comparable to high-NFE baselines with substantially fewer NFEs. We therefore omitted our high-NFE results from the original figure. We have now revised Figure 4 to include GLSM results at 512 and 1024 NFEs. The table below further compares GLSM with NCSN++ (PC) up to 2048 NFEs, showing that GLSM still maintains an advantage.

CIFAR-10 FID ↓\downarrow25651210242048
GLSM (ours)6.475.074.534.44
NCSN++ (PC)12.15.584.584.58

(Q4) Class-wise evaluation and support coverage. To ensure a fair comparison, we use the same standard frozen classifier and evaluation protocol for all methods (Appendix D.4.1). We also evaluate recall and coverage independently of classifier predictions, providing direct evidence of how well each method recovers the data support (Appendix D.4.2, Tables 11-12). At 128 NFEs, GLSM achieves the highest recall (0.687 vs. 0.603), coverage (0.868 vs. 0.779), and Tail-3 coverage (0.967 vs. 0.845), compared with the strongest competing result for each metric. Thus, our improvements are also confirmed by classifier-independent measures of distributional coverage.

AuthorsOpenReview ↗

Response to Reviewer Wufs — Part 3/3

(Q5) Generality of the Lévy measure. We further evaluated gamma and α\alpha-stable jumps on synthetic and image-generation benchmarks (Table 1, Figure 2, and Table 5).

Target (NFE)MetricSMLDGLSM-CPGLSM-α\alphaGLSM-Γ\Gamma
Cauchy (32)Sliced W1↓W_1\downarrow1.11300.11800.40000.1650
Tail MSLE ↓\downarrow0.07180.00190.00580.0009
GMM (128)Sliced W1↓W_1\downarrow0.02870.01630.00620.0198
Mode-TV ↓\downarrow0.04140.02440.01020.0381
Method (128 NFEs)CIFAR-10 FID ↓\downarrowCIFAR-10-LT FID ↓\downarrowLT class-TV ↓\downarrow
NCSN++ (PC)59.5050.700.307
GLSM-CP10.0710.130.135
GLSM-α\alpha (α=1.5\alpha=1.5)15.6612.360.157
GLSM-α\alpha (α=1.0\alpha=1.0)11.209.500.141

These additional Lévy measures achieve performance comparable to (even slightly better than) original compound-Poisson jumps, while consistently outperforming the Brownian baselines. These results show that our sampling improvements are not specific to a particular Lévy measure.

(Q6) Matched comparison with LIM. We thank the reviewer for this suggestion. To control for differences in architecture and training budget, we conducted additional experiments comparing GLSM-α\alpha with LIM on the two-dimensional Cauchy and imbalanced Gaussian-mixture datasets. Both methods use the same FieldNet architecture (830,723 parameters), optimizer, training budget (40,000 steps with batch size 8192), and isotropic α\alpha-stable noise with the same α\alpha and numerical clipping convention. LIM retains its original forward process, learning objective, and reverse sampler, whereas GLSM uses Gaussian forward perturbations and the Lévy Langevin corrector.

The results at the settings used in our manuscript are summarized below (sliced W1↓W_1\downarrow, 50,000 samples over 10 seeds):

TargetNFEα\alphaGLSM-α\alphaLIM
Cauchy321.50.39970.7852
GMM1281.70.00620.0117

Under this strictly controlled setting, GLSM-α\alpha achieves nearly 2×2\times lower sliced W1W_1 than LIM on both benchmarks at the settings reported in our manuscript. Specifically, GLSM-α\alpha reduces the error from 0.78520.7852 to 0.39970.3997 on Cauchy and from 0.01170.0117 to 0.00620.0062 on the Gaussian mixture. The improvement is even more pronounced on Cauchy at 64 NFEs with α=1.5\alpha=1.5, where GLSM-α\alpha achieves a 5.73×5.73\times reduction in sliced W1W_1. Importantly, these gains are obtained with identical network architectures, training budgets, and stable indices. We further verified that retraining LIM with its original MLP under the matched training budget yields nearly identical performance to the shared FieldNet architecture. These results demonstrate that the observed improvements cannot be attributed to greater network capacity or additional training, providing strong empirical support for the effectiveness of our Gaussian-forward Lévy generative framework.

Reviewer_BxYuOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes two components of Levy-based diffusion models: (i) denoising Levy score matching (DLSM), which learns Levy score from Gaussian-perturbed data; and (ii) Levy Langevin correctors (LLCs), which produce samples from the target distribution by using the scores learned by DLSM. By combining DLSM and LLCs, the authors propose Gaussian–Levy Score Model (GLSM). Numerical experiments show that GLSM improves the mode-density recovery and sample quality.

Critical strengths
  • The proposed algorithm is well motivated, and experiments on synthetic data, CIFAR, and Celeba show better mode coverage and image quality compared to previous methods.
Critical weaknesses
  • The experiments did not separately validate the use of LLC and DLSM. This paper could benefit from ablation studies of the two components.
Questions for authors
  • It may be better to highlight Gaussian–Levy Score Model (GLSM) as a main contribution in the abstract.
  • Figure 4 shows that at NFE=32 or 64, FID of GLSM could be larger than that of DLPM, and it does not show the results of GLSM at NFE=512 or 1024.
  • The results reported in Figure 4 do not match Table 5.
4 comments
AuthorsOpenReview ↗

Response to Reviewer BxYu — Part 1/2

We thank the reviewer for recognizing the strengths of our method and for the constructive suggestions. We address the concerns below.

(W1) Validate the use of LLC and DLSM separately with ablation studies. We thank the reviewer for this comment. We realize that the roles of DLSM and LLC may not have been explained sufficiently clearly in our presentation.

DLSM is used for training: learn the Lévy score from Gaussian-perturbed data, while LLC is used for sampling: use the Lévy score from DLSM to derive a jump-type generative process. So they are not separate independent components in our algorithm.

We take analogy of conventional Brownian score-based generative models to illustrate the relation between DLSM and LLC:

TrainingSampling
Brownian modelsDSM (Stein score)Annealed Langevin Corrector
GLSM (ours)DLSM (Lévy score)Annealed Lévy Langevin Corrector

Therefore, without DLSM, there is no Lévy score Sν(t,x)S_\nu(t,x) for LLC to generate samples; without adopting LLC during sampling, there is no need to apply DLSM for Lévy score. Due to this sequential coupling, removing either component would make the model dysfunctional rather than providing meaningful ablations.

We have clarified this point in Section 3.1 (lines 247-253) and Section 3.3 (lines 308-339).

AuthorsOpenReview ↗

Response to Reviewer BxYu — Part 2/2

(Q1) Highlighting GLSM in the abstract. We thank the reviewer for the suggestion. We have revised the opening of the abstract (lines 011) to explicitly introduce the GLSM as our main framework, with DLSM and LLC as its learning and sampling components, respectively.

(Q2) Performance across NFEs in Figure 4. We first supplement Figure 4 by adding the results at 512, 1024 and 2048 NFEs for our method and the strongest baseline, NCSN++ (PC), summarized in the table below.

CIFAR-10 FID ↓\downarrow25651210242048
GLSM (ours)6.475.074.534.44
NCSN++ (PC)12.15.584.584.58

The FIDs at these high NFEs well match the convergence trend of the original data in Figure 4 and we shall update Figure 4 accordingly.

At the lowest NFE=32 and 64, we do observe that DLPM achieves the best FID which is close to 20. However as the original paper of DLPM (their Table 3) showed, the FID actually does not decrease further below 10, even for NFE as large as 1000. This phenomenon is further confirmed here in our Figure 4. Therefore, it seems that DLPM is weak to achieve the arbitrary low FID as the NFE increases. It is also noted that LIM and DLPM share the almost identical saturated FID at the high NFEs (≥256\ge 256). We do not have a full understanding of this pattern for DLPM and LIM; but we conjecture that since both used α\alpha-stable noise, their requirement of a practical truncation as an approximation might trigger that the final error measured by FID may not shrink towards zero.

Nevertheless, the general requirement for a useful generative method should have FID below 10, while the existing α\alpha-stable-noise-based models like LIM and DLPM are not observed to achieve this basic tolerance regardless of a very large NFE invested. The out-performance of DLPM in such low NFE region requires some further understanding but has been shown not generalizable to large NFE.

(Q3) Difference between Figure 4 and Table 5. When calculating the FID, Figure 4 and Table 5 (now Table 6 in revision) actually used different number of generated samples: 10,000 generated samples per seed (three seeds for CIFAR-10), vs. 2,048 generated samples from a single seed. These details have been well documented in Appendix D.2.3, lines 1004-1012.

The reason why we used different resolutions in numerical calculation of FID is that Figure 4 and Table 5 (Table 6 in the revised manuscript) serve different purposes. Figure 4 in our main body compares our method with existing generative models at matched NFEs, whereas Table 6 is only an ablation study to examine the effect of the number of annealing levels for our own method (Appendix D.3.1, lines 1020-1025).

I thank the authors for the detailed response, which clarifies most of my concerns. I raised my score accordingly.

Thank you for your positive feedback and for reconsidering your score. We sincerely appreciate your time and constructive comments.

Comments on the paper

1 comment

Dear Reviewers, The authors have now submitted their responses to the reviews. There is only a limited amount of time left for you to review their replies and provide any further comments.

Your feedback is crucial to ensure that the review process proceeds smoothly. We would greatly appreciate it if you could share your reactions before the discussion period closes.

Thank you very much for your work. Best regards.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.