acceptodds
Under review as a conference paper at ICLR 2027

Tail-Index Annealing for Long-Tailed Diffusion Models

Abstract

Classical denoising diffusion models rely on Gaussian noise throughout the forward process. Recent stable-noise diffusion models, such as the Denoising Lévy Probabilistic Model (DLPM), replace it with stable noise that has a single fixed tail index, forcing a fixed compromise between terminal initialization and reverse estimation. Direct substitution of a time-varying tail index into DLPM fails since increments with different indices do not form a single stable law. We introduce **Annealed DLPM** (A-DLPM), a consistent framework that increases the tail index along a half-cosine forward schedule and traverses the reverse schedule during generation. We construct the corresponding conditional Gaussian reverse kernel, which reduces to the original DLPM when the tail index is fixed. The resulting annealed residual admits an exact, interpretable spectral-curvature characterization. On two-dimensional data, A-DLPM attains the lowest mean tail-sensitive error among evaluated methods; on long-tailed image benchmarks, it attains the highest mean Recall among all evaluated methods and improves both FID and Recall over the evaluated DLPM, DLIM, and Gaussian DDPM controls.

Primary area:
generative models
Keywords:
Diffusion Models, Heavy-Tailed Distributions, Stable Laws, Long-tailed Data, Generative Modeling

Reviews

Among:
Shape:
1234this paper, Top 32%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.50 over 4 reviews, above 68% of the others.
Reviewer_jkLROpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

Paper Summary

This paper generalizes DLPM by letting the stable tail index αt\alpha_t vary across timesteps: small α\alpha near the data end, annealed to α=2\alpha = 2 at the terminal end. Generation reverses this schedule — starting near-Gaussian like DDPM, ending with heavy-tailed α=1.5\alpha = 1.5, whose large jumps help cover rare modes in long-tailed data. Since stable increments with different tail indices are not additive-closed, the authors condition on the full scale path A0:tA_{0:t} and introduce a signal-correction term plus a variance recursion, keeping the forward marginal Gaussian and training cheap. They provide complete training and sampling algorithms, and an effective spectral index characterizing the noise's tail behavior per frequency — heavy-tailed at low frequencies (αmin⁡\alpha_{\min}), near-Gaussian at high frequencies (αmax⁡\alpha_{\max}).

Critical strengths

Strengths

  1. Sound and rigorous theory within the established paradigm. The methodology follows the DDPM/DLPM line of work. And all derivations and proofs (forward marginals, reverse kernels, distribution characterization) are clear and, to my checking, correct.

  2. A faithful and complete generalization of DLPM. The paper generalizes DLPM end-to-end, from the forward process design (signal-correction term and variance recursion accommodating time-varying tail indices) to the reverse sampling procedure (arbitrary-jump conditional-Gaussian kernels with respacing support). When the tail index is constant, the framework reduces exactly to DLPM (Prop. 8), making the generalization conservative and verifiable.

  3. Clear organization and presentation. The paper is well-structured and logically coherent: the problem statement is precise (four criteria for a valid construction, each addressed in turn), the writing is fluent, and the extensive appendix (proofs, algorithms, ablations, sensitivity analyses) makes the work easy to verify and reproduce.

Critical weaknesses

Weaknesses

  1. Convergence of the underlying framework remains open. The only sampling guarantee for fixed-α\alpha DLPM (Cherkaoui et al., 2026, Thm. 2.2) assumes uniform approximation of the reverse object by an idealized TV-risk minimizer, whereas practical epsilon-RMSE training yields only L2(pt)L^2(p_t)-type control — and the guarantee does not extend to the time-varying construction here. We encourage the community to prioritize these foundational convergence questions.

  2. A theory–practice gap in conditioning. The reverse kernel is exact only given the realized scale path, yet the network sees only (Xt,t)(X_t, t) and learns a marginalized predictor; how this predictor interacts with the path-conditioned kernel coefficients at generation is not analyzed.

  3. Unanalyzed plug-in approximation. Sampling substitutes X^0,θ\hat{X}_{0,\theta} and resampled scales into a kernel that is exact only for the true X0X_0 and scale path; the resulting error is neither bounded nor empirically isolated.

  4. Theory does not cover the implemented object. Thm. 1/2 and Prop. 9 assume untruncated stable scales, but the implementation clips At≤20A_t \leq 20, after which increments are no longer stable; results are sensitive to the threshold and no truncated theory is given.

Questions for authors

Questions For Authors

  1. Convergence. Can any distribution-level sampling guarantee (even under strong assumptions) be established for the time-varying tail-index construction? If not, which step of the Cherkaoui et al. (2026) framework breaks down most fundamentally — the non-Markovianity after marginalizing X0X_0, the mixed-power residuals, or the scale-augmented epsilon target?

  2. Conditioning gap. The network is trained marginally over scale paths (input only (Xt,t)(X_t, t)), while the reverse kernel's coefficients are computed from the realized path. Can you quantify — theoretically or empirically — how much sampling error comes from this mismatch, e.g., by comparing against an oracle that also receives the true scales?

  3. Plug-in error. How sensitive is generation quality to the accuracy of X^0,θ\hat{X}_{0,\theta}? For instance, does an oracle sampler using the true X0X_0 (on synthetic data where it is available) close a significant fraction of the gap to the target distribution?

  4. Clipping. Do truncated analogues of Thm. 1/2 and Prop. 9 exist (e.g., tail behavior governed by αmin⁡\alpha_{\min} up to the clip level)? How do the main results vary with the clipping threshold, and is At≤20A_t \leq 20 needed uniformly across all experiments?

8 comments

We thank the reviewer for their thoughtful feedback and for recognizing the strengths of our work. Below, we address the reviewer's concerns in detail.

Q1. Convergence. Can any distribution-level sampling guarantee (even under strong assumptions) be established for the time-varying tail-index construction? If not, which step of the Cherkaoui et al. (2026) framework breaks down most fundamentally — the non-Markovianity after marginalizing X0X_0, the mixed-power residuals, or the scale-augmented epsilon target?

The main obstacle to transferring this framework is controlling the local TV errors of the implemented reverse transitions. In particular, the deterministic final denoising step creates a maximal local TV discrepancy, regardless of the prediction accuracy.Our epsilon-RMSE objective trains the network to predict a point estimate, whereas the analysis in Cherkaoui et al. [1] (Theorem 2.2 and Appendix C.2). controls the TV distance between reverse transition distributions.

Cherkaoui et al. derive their sampling bound by decomposing the total-variation (TV) error into terminal-initialization and reverse-transition errors, then controlling the latter through a TV-based learning objective. Their decomposition applies to Markov path laws. In A-DLPM, the forward transition in Eq. (7) explicitly depends on X0X_0, so the process is generally non-Markovian after marginalizing X0X_0. Nevertheless, a corresponding decomposition remains valid for the final marginal distribution. Following the notation of [1], with aa denoting the realized scale path and α\alpha the tail-index schedule, we have

= \int k_{t-1\mid t,a}^{(\alpha)}(B\mid x) \, k_{t\mid a}^{(\alpha)}(\mathrm dx), \qquad \forall B\subset\mathbb{R}^d\text{ Borel}$$ Here $k_{s\mid t,a}^{(\alpha)}(x)=\mathcal L(X_s\mid X_t=x,A=a)$ is the exact reverse probability law. Let $\overleftarrow q_{s\mid t,a}^{\theta}(\cdot\mid x)$ be its implemented counterpart: the Gaussian in Proposition 7 with $X_0$ replaced by $\widehat X_0(x,t)$. For retained times $0=t_0<\cdots<t_N$, let $\widehat k_{t_i\mid a}^{\theta}$ denote the sampler marginal. Eq. (8) of [1] gives the following TV error decomposition: $$\begin{aligned} \mathrm{TV}\\!\left(\widehat k_{0\mid a}^{\theta},k_{0\mid a}^{(\alpha)}\right) &\leq \mathrm{TV}\\!\left(\widehat k_{t_N\mid a}^{\theta},k_{t_N\mid a}^{(\alpha)}\right)\\\\ &\quad+\sum_{i=1}^{N}\int \mathrm{TV}\\!\left( k_{t_{i-1}\mid t_i,a}^{(\alpha)}( \cdot \mid x), \overleftarrow q_{t_{i-1}\mid t_i,a}^{\theta}( \cdot \mid x) \right)k_{t_i\mid a}^{(\alpha)}(\mathrm dx). \end{aligned}$$ The first term is the terminal-initialization error; each transition error is averaged over the true conditional forward marginal. Averaging the right-hand side over the common scale-path law also bounds the unconditional output TV error, by convexity of TV. The exact reverse kernel averages over the posterior of $X_0$, whereas the implemented kernel uses a point estimate. At the final step, $t=t_1$ to $0$, they become $$\begin{aligned} k_{0\mid t,a}^{(\alpha)}(x) &=\mathcal L(X_0\mid X_t=x,A=a)\\\\ \overleftarrow q_{0\mid t,a}^{\theta}(x) &=\delta_{\widehat X_0(x,t)} \end{aligned}$$ Suppose the data distribution has a density and the forward conditional noise variance is positive. The posterior of $X_0$ then assigns zero probability to every singleton. Taking $B=\\{\widehat X_0(x,t)\\}$, the implemented kernel assigns probability one to $B$, while the exact kernel assigns probability zero. Hence, $$\mathrm{TV}\\!\left( k_{0\mid t,a}^{(\alpha)}( \cdot \mid x), \overleftarrow q_{0\mid t,a}^{\theta}(\cdot \mid x) \right)=1.$$ The final-step term alone therefore makes the upper bound at least one, regardless of prediction accuracy. Since TV is at most one, this bound is uninformative for the implemented sampler. Regarding the three possible obstacles, non-Markovianity prevents direct use of the complete-path decomposition but does not invalidate marginal-level error propagation. The mixed-power residuals require new estimates of the terminal-initialization error, since the fixed-index stable noise law in [1] is replaced by the mixed-power characteristic exponent in Eq. (18). The more fundamental difficulty lies in the learned reverse transitions. The epsilon-RMSE objective controls prediction error rather than the TV distance between conditional distributions. In particular, the deterministic final update makes the corresponding local TV error equal to one under a non-atomic data distribution, regardless of prediction accuracy. A nontrivial distribution-level guarantee therefore requires a different way to control the actual sampler's output distribution, together with a new initialization analysis for the mixed-power residuals.

Q2. Conditioning gap. The network is trained marginally over scale paths (input only (Xt,t)(X_t,t)), while the reverse kernel's coefficients are computed from the realized path. Can you quantify — theoretically or empirically — how much sampling error comes from this mismatch, e.g., by comparing against an oracle that also receives the true scales?

We can quantify the effect of omitted scale information on endpoint prediction and on an individual reverse update. In the two balanced synthetic mixtures below, the comparison between the marginal and scale-conditioned conditional-mean references shows no consistent generation-quality advantage from observing the scales.

Algorithm 2 in Appendix E samples a complete scale path AA before the reverse loop and retains it for the reverse coefficients. The network receives only (Xt,t)(X_t,t). Its endpoint estimate is the quantity already defined in Eq. (11):

\frac{X_t-\bar\sigma_t\epsilon_\theta(X_t,t)} {\bar\gamma_t}.$$ At a fixed time $t$, define the two conditional-mean references under the true forward joint distribution: $$m=\mathbb E[X_0\mid X_t],\qquad m_A=\mathbb E[X_0\mid X_t,A].$$ The first uses the same information as the network; the second additionally observes the realized scale path. For data with finite second moments and an endpoint predictor with finite squared error, conditional expectation gives $$\mathbb E\\|\widehat X_{0,\theta}-m_A\\|^2 = \mathbb E\\|\widehat X_{0,\theta}-m\\|^2+ \mathbb E\\|m-m_A\\|^2.$$ The cross term vanishes because $\mathbb E[m_A\mid X_t]=m$ and $\widehat X_{0,\theta}$ depends only on $(X_t,t)$. In particular, the second term remains even if the endpoint predictor equals $m$. It also equals the reduction in optimal endpoint MSE obtained by observing the scales, $\mathbb E\\|X_0-m\\|^2-\mathbb E\\|X_0-m_A\\|^2$. For a retained pair $s<t$, let $a=a_{s,t}$ and $b=b_{s,t}$ be the coefficients in Eq. (14), and let $\nu=\nu_{s,t}>0$ be the variance in Eq. (15). At a fixed state and realized scale path, replacing the actual endpoint estimate by $m_A$ changes the plug-in Gaussian reverse kernel by $$D_{\mathrm{KL}}\left( \mathcal N(aX_t+b\widehat X_{0,\theta},\nu I) \\,\Vert\\, \mathcal N(aX_t+bm_A,\nu I) \right)= \frac{b^2}{2\nu}\\|\widehat X_{0,\theta}-m_A\\|^2.$$ Setting $\widehat X_{0,\theta}=m$ in this expression isolates the local change due to omitted scale information. The factor $b^2/(2\nu)$ depends on the realized path, so the unweighted squared-error decomposition above does not automatically decompose the expected KL. Empirically, we compared the learned endpoint estimate with the analytic $m_A$ on a known two-dimensional Gaussian mixture. We also approximated $m$ by randomized quasi-Monte Carlo integration over the prior scale-path law with posterior likelihood weights. All predictors use the same plug-in Gaussian update, including the same conditional variance. The diagnostic uses A-DLPM checkpoints trained on the two-dimensional mixtures across five training seeds, with five generation seeds per checkpoint and 5,000 samples per generation. The conditional-mean references have no trained parameters and use the same five generation seeds. Predictors share initialization, scale paths, and Gaussian innovations within each generation seed. Both mixtures consist of equally weighted isotropic Gaussian components ($\sigma=0.1$). Entries are mean sliced-W1; lower is better. | Scale cap | Separation | Original predictor | Refined predictor | Marginal reference $m$ | Scale-conditioned reference $m_A$ | | --- | --- | --- | --- | --- | --- | | Untruncated | $4\sigma$ | .01758 | .01101 | .01955 | .02041 | | Untruncated | $16\sigma$ | .03251 | .01518 | .01999 | .01931 | | $C=20$ | $4\sigma$ | .01166 | .00634 | .01068 | .01216 | | $C=20$ | $16\sigma$ | .02042 | .01258 | .01345 | .01525 | The refined predictor achieves lower mean sliced-W1 than both conditional-mean references across all four settings. Comparing the two conditional-mean references, $m_A$ gives a lower mean sliced-W1 only in the untruncated $16\sigma$ setting, whereas $m$ gives lower values in the other three. Thus, providing true scale information yields no consistent full-generation advantage under the fixed plug-in update. We also observed a similar phenomenon in our early experimental design: feeding the scale $A_t$ to the network as an additional input leads to a clear degradation in performance, which is also reported by Appendix G of DLPM paper [1]. [1] Dario Shariatian, Umut Simsekli, and Alain Durmus. Denoising Lévy Probabilistic Models. *ICLR, 2025*. https://arxiv.org/abs/2407.18609

Q3. Plug-in error. How sensitive is generation quality to the accuracy of X^0,θ\hat X_{0,\theta}? For instance, does an oracle sampler using the true X0X_0 (on synthetic data where it is available) close a significant fraction of the gap to the target distribution?

Reducing endpoint prediction error improves individual reverse updates, and a posterior oracle substantially lowers mean error in full generation on the known Gaussian mixture.

At a fixed state and scale path, let a=as,ta=a_{s,t} and b=bs,tb=b_{s,t} be the coefficients in Eq. (14), and ν=νs,t>0\nu=\nu_{s,t}>0 be the variance in Eq. (15). Proposition 7 gives the conditional Gaussian update in Eq. (12). Replacing the true endpoint by the network prediction changes the reverse mean by b(X^0,θ−X0)b(\widehat X_{0,\theta}-X_0), giving

\mathcal N(aX_t+bX_0,\nu I) \\,\Vert\\, \mathcal N(aX_t+b\widehat X_{0,\theta},\nu I) \right) =\frac{b^2}{2\nu}\\|\widehat X_{0,\theta}-X_0\\|^2.$$ Consequently, this KL divergence of the reverse update exhibits a quadratic dependence on the endpoint prediction error. Eq. (11) also gives $\widehat X_{0,\theta}-X_0=(\bar\sigma_t/\bar\gamma_t)(\epsilon_t-\epsilon_\theta)$, linking this sensitivity directly to epsilon prediction. We tested sensitivity using $X_0+\lambda(\widehat X_{0,\theta}-X_0)$, with $\lambda\in\\{0,1/2,1\\}$, while holding the forward state, scale path, and Gaussian innovation fixed. Across four mixture settings and $t\in\\{25,50,75,99\\}$, halving the endpoint error reduced the mean sliced-W1 (relative to samples from the true endpoint) by 36.9%–49.9%. Each comparison averages three repetitions. This measures the effect on samples after one reverse step; the quadratic formula above concerns conditional KL. We use positive-variance steps, since supplying the true endpoint at the final deterministic step simply returns that data point. For full generation, since the ground-truth endpoint $X_0$ is unknown, the oracle instead draws $X_0\sim p(X_0\mid X_t,A)$ at each step and applies the conditional Gaussian update, implementing the exact reverse conditional. It shares terminal initialization, scale paths, and Gaussian innovations with the learned sampler, with a separate stream for posterior draws. We compare the original predictor with a refined predictor trained with additional updates (15,000 for untruncated scales, 5,000 for $C=20$) selected via independent validation samples. Both predictors receive $(X_t,t)$ and use the same epsilon-RMSE objective. The results below report the balanced mixture settings. Each learned configuration uses five paired training checkpoints and five generation seeds per checkpoint, with 5,000 samples per generation. We average over generation seeds, then checkpoints. The oracle uses the same five generation seeds and has no trained parameters. Entries are mean sliced-W1 to a common target reference; relative reductions use unrounded means. | Scale cap | Separation | Original predictor | Refined predictor | Posterior oracle | Relative reduction from refined predictor | | --- | --- | --- | --- | --- | --- | | Untruncated | $4\sigma$ | .01758 | .01101 | .00454 | 58.7% | | Untruncated | $16\sigma$ | .03251 | .01518 | .00806 | 46.9% | | $C=20$ | $4\sigma$ | .01166 | .00634 | .00341 | 46.1% | | $C=20$ | $16\sigma$ | .02042 | .01258 | .00627 | 50.1% | The refined predictor achieves lower mean sliced-W1 than the original predictor in all four settings. For the capped process, its endpoint MSE, averaged over all 99 positive forward times, is also lower. The posterior oracle further reduces mean sliced-W1 by 46%–59% relative to the refined predictor. This comparison reflects the combined benefit of conditioning on the realized scale path and sampling from the conditional posterior.
Reviewer_5fZ1OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper extends DLPM [1], α-stable diffusion with a fixed tail index, to a step-dependent index rising from 1.5 at the data to 2.0 at the terminal along a half-cosine. Conditioning on the sampled scale path of the Gaussian scale mixture keeps the forward marginal and the reverse kernel Gaussian, so DLPM's ε-regression objective is retained; a signal-correction term fixes the cumulative signal coefficient at q_t^{1/α_t}. The residual's characteristic exponent and a continuous-time limit are derived. Experiments on a 2-D toy, CIFAR-10-LT and CIFAR-100-LT compare against DDPM, DLPM, DLIM and LIM at six fixed tail indices.

Critical strengths

The construction is exact given the scale path (forward marginal, reverse kernel, respacing) and reduces to DLPM at fixed α (Prop. 8), so it is derived rather than heuristic. The evaluation hygiene is better than usual: five training seeds times five generation seeds, paired confidence intervals, and six tail indices for each baseline family. The sensitivity studies are informative in themselves: heavy tails at the high-noise end are very harmful, and ending the path at 1.9 instead of 2.0 costs about 2 FID.

Critical weaknesses

My main concern is that the empirical case is not established. On CIFAR-10-LT the gain over DDPM is 1.3 FID. On CIFAR-100-LT, LIM at α = 1.8 has significantly lower FID (−1.70 [−2.42, −0.98]) with indistinguishable Recall, so the statement that LIM "trades lower FID for lower Recall" is not supported by the paper's own intervals. More importantly, the baselines look under-trained. The protocol matches [1] except that it runs 75k instead of 400k updates. Here DLPM is worst at α = 1.5 (23.45) and best at α = 2.0 (19.00), whereas [1] reports 16.10 at 1.5 and 21.07 at 2.0, and A-DLPM's best FID (17.30) does not reach the published fixed-α number. A-DLPM's own FID worsens between 75k and 100k updates (17.2 → 18.7 and 27.2 → 34.9, Table 4) while no baseline trajectory is shown, so the comparison may reflect convergence speed rather than final quality. Finally, the FID/PRDC reference set is drawn from the training pool in training proportions, so the headline metric rewards reproducing the imbalance rather than covering the tail; Fig. 4 omits DDPM and LIM, and on CIFAR-100-LT A-DLPM beats all six DLPM means in per-class Coverage on only 51 of 100 classes. The standard CIFAR-LT diffusion baseline CBDM [2] is neither cited nor compared.

Second, the methodological contribution is simpler than the presentation suggests, and the added machinery is not shown to matter. Drawing A_t with index α_t/2 in DLPM's augmented chain already gives a valid forward chain, an exact reverse kernel and an ε-target; A-DLPM differs only in the convention γ̄_t = q_t^{1/α_t}, which forces the X_0-dependent correction, the non-Markovianity and the drift in Eq. (20). The only stated motivation is the reduction to DLPM at constant α, which the Markov product Π_k γ_k also satisfies, and the abstract's claim that direct substitution "fails" is never run. Theorems 1, 2 and 13 do not enter the algorithm and describe the untruncated process, whereas the implemented model clips A_t at 20 and degrades sharply without clipping (Table 9). No ablation isolates the mechanism. Both concerns are addressable with converged baselines and the controls asked for below.

I did not check the continuous-time limit in Appendix D line by line.

[1] Shariatian et al. Denoising Lévy Probabilistic Models. ICLR 2025. [2] Qin et al. Class-Balancing Diffusion Models. CVPR 2023.

Questions for authors
  1. Please show FID and Recall trajectories to convergence for DLPM, LIM and DDPM on CIFAR-10-LT, and explain why the α-ordering of DLPM in Table 1 is the reverse of that in [1].
  2. Please run DLPM's augmented chain with A_t ~ A_{α_t/2} and γ̄_t = Π_k γ_k (no correction), which is the natural "direct substitution". Is there a motivation for the q_t^{1/α_t} convention beyond the reduction at constant α?
  3. Please add a Gaussian control with A_t ≡ 2 and the same (γ̄_t, σ̄_t) schedule, to separate the changed signal schedule from the heavy tails.
  4. Did the published DLPM results [1] use scale clipping? Given Table 9, does the gain come from a bounded scale mixture rather than from heavy tails?
10 comments

We thank the reviewer for the thoughtful feedback and for highlighting important points for discussion. We address them below.

Q1. Please show FID and Recall trajectories to convergence for DLPM, LIM and DDPM on CIFAR-10-LT, and explain why the α-ordering of DLPM in Table 1 is the reverse of that in [1].

Under our protocol, extending training to 400k updates yields substantial deterioration in FID and Recall across all six configurations, rather than a stable generation-quality plateau. This is consistent with overfitting, and these trajectories do not establish convergence.

The original Appendix H.1, Figure 7 and Table 4 report A-DLPM's trajectory. We have now extended the comparison to Gaussian DDPM, LIM at α=1.8\alpha=1.8, and DLPM at α=1.5,1.8,2.0\alpha=1.5,1.8,2.0. The table reports FID / Recall, using one training run per method, one shared generation seed, 5,000 generated images per checkpoint, and the same long-tailed reference throughout. Each method attains its lowest sampled FID at 75k or 100k.

Method75k100k150k300k400k
A-DLPM17.491 / .648618.968 / .655626.079 / .620038.803 / .536641.635 / .5190
DLPM, α=1.5\alpha=1.523.630 / .617423.232 / .621226.550 / .613636.407 / .586439.254 / .5666
DLPM, α=1.8\alpha=1.819.537 / .634620.806 / .627827.374 / .599041.066 / .567844.228 / .5602
DLPM, α=2.0\alpha=2.019.272 / .635422.256 / .620032.313 / .576451.790 / .546856.086 / .5482
Gaussian DDPM18.771 / .641222.100 / .612631.279 / .567246.630 / .475249.593 / .4698
LIM, α=1.8\alpha=1.819.938 / .651418.809 / .654624.061 / .620835.779 / .559838.747 / .5394

Table 1 reports averages over five training runs and five generation seeds per run at a fixed budget of approximately 75k updates, so its values differ from the individual trajectories shown here.

DLPM at α=1.5\alpha=1.5 overtakes α=2.0\alpha=2.0 between 100k and 150k because its FID increases more slowly during this period. The reversal therefore reflects different rates of deterioration under extended training. At 400k, the FID ordering is α=1.5<1.8<2.0\alpha=1.5 < 1.8 < 2.0, matching Table 3 of Shariatian et al. [1], whose CIFAR-10-LT models were also trained for 400k updates. This explains how training duration can reverse the fixed-budget ranking within our protocol, without establishing an ordering at convergence.

Numerical comparisons also depend on implementation and evaluation choices: Appendix G.2 of [1] reports 5,000 real reference images without specifying how they were selected, whereas the CIFAR-10-LT configuration in their GitHub release defaults to the full training pool of 12,406 images. Our experiments use a fixed reference of 5,000 images that preserves the long-tailed class proportions.

[1] Dario Shariatian, Umut Simsekli, and Alain Durmus. Denoising Lévy Probabilistic Models. ICLR, 2025.

Q2. Please run DLPM's augmented chain with At∼Aαt/2A_t\sim\mathcal A_{\alpha_t/2} and γˉt=∏kγk\bar\gamma_t=\prod_k\gamma_k (no correction), which is the natural "direct substitution". Is there a motivation for the qt1/αtq_t^{1/\alpha_t} convention beyond the reduction at constant α\alpha?

Our construction uses the prescribed signal attenuation. The augmented chain proposed in the review gives a different cumulative signal coefficient. Section 3.3, Eqs. (4)–(8), and Appendix B.1 specify that construction. With qt=∏j≤t(1−βj)q_t=\prod_{j\leq t}(1-\beta_j), A-DLPM sets γˉt=qt1/αt\bar\gamma_t=q_t^{1/\alpha_t} and σˉt=(1−qt)1/αt\bar\sigma_t=(1-q_t)^{1/\alpha_t}, preserving γˉtαt+σˉtαt=1\bar\gamma_t^{\alpha_t}+\bar\sigma_t^{\alpha_t}=1. Given this prescribed noise coefficient, qt1/αtq_t^{1/\alpha_t} is the unique nonnegative signal coefficient satisfying that normalization at each time, extending the coefficient convention of fixed-index diffusion to the current index αt\alpha_t.

For the uncorrected chain, denote its cumulative signal coefficient by γ~t:=∏j≤t(1−βj)1/αj\widetilde\gamma_t:=\prod_{j\leq t}(1-\beta_j)^{1/\alpha_j}. Under an increasing index schedule,

=\sum_{j=1}^{t}\left(\frac{1}{\alpha_j}-\frac{1}{\alpha_t}\right) \log(1-\beta_j)\leq0.$$ Thus, the uncorrected chain accumulates greater signal attenuation than the prescribed schedule. The correction compensates for this additional attenuation. With the same scales and Gaussian innovations, the two constructions share the same residual recursion and differ only in the signal path. This motivates the correction beyond the constant-index case and gives a direct ablation of the signal path. We trained the construction without the correction using its corresponding forward marginals and reverse kernels. The table reports means over three paired training runs, with one shared generation seed per model. Coverage uses the overall long-tailed reference. | Dataset | A-DLPM FID | No correction FID | A-DLPM Recall | No correction Recall | A-DLPM Coverage | No correction Coverage | | --- | --- | --- | --- | --- | --- | --- | | CIFAR-10-LT | 17.2863 | 17.0837 | .6583 | .6557 | .8937 | .8896 | | CIFAR-100-LT | 27.3242 | 26.4747 | .6393 | .6396 | .8215 | .8320 | The FID differences remain unresolved and Recall means are close. Coverage favors the correction on CIFAR-10-LT, while its difference on CIFAR-100-LT remains unresolved. The correction therefore does not show a consistent empirical advantage across metrics. We emphasize that its role is mathematical: it implements the prescribed signal attenuation, and the ablation tests exactly this role. The abstract's statement that direct substitution fails concerns the fixed-index stable-marginal formulas. As Theorem 1 establishes, distinct active indices yield $\Psi_t(r)=\sum_jw_{j,t}^{\alpha_j}r^{\alpha_j}$, which cannot be represented by one stable exponent $Cr^\alpha$. Conditioning on the realized scale path yields an exact chain, and we derive its accumulated variance and reverse coefficients. Our contribution supplies this scale-conditioned construction, arbitrary-jump conditional reverse updates, and a distributional characterization of the resulting noise.

Q3. Please add a Gaussian control with At≡2A_t\equiv2 and the same (γˉt,σˉt)(\bar\gamma_t,\bar\sigma_t) schedule, to separate the changed signal schedule from the heavy tails.

We ran the requested At≡2A_t\equiv2 control using the cumulative coefficients in Eq. (4). A-DLPM has lower mean FID and higher mean Recall on both datasets in this comparison. Under the stable-scale convention in Section 3.1, S2=N(0,2I)\mathcal S_2=\mathcal N(0,2I), the control's marginal is

Xt∣X0∼N(γˉtX0,2σˉt2I).X_t\mid X_0\sim\mathcal N(\bar\gamma_tX_0,2\bar\sigma_t^2I).

The step coefficients are recomputed as gt=γˉt/γˉt−1g_t=\bar\gamma_t/\bar\gamma_{t-1} and st2=σˉt2−gt2σˉt−12s_t^2=\bar\sigma_t^2-g_t^2\bar\sigma_{t-1}^2, giving Xt=gtXt−1+st2ZtX_t=g_tX_{t-1}+s_t\sqrt2 Z_t. Training, reverse kernels, and terminal initialization use this same process. This matches the nominal cumulative schedule, with a deterministic variance. By contrast, A-DLPM's variance is random and depends on the sampled scale path.

DatasetA-DLPM FIDGaussian control FIDA-DLPM RecallGaussian control Recall
CIFAR-10-LT17.286317.8890.6583.6498
CIFAR-100-LT27.324227.8573.6393.6223

Across the three training seeds, the controlled means favor A-DLPM in both FID and Recall on both datasets. With the nominal cumulative schedule held fixed, these results suggest a contribution from the annealed stable-noise construction, which changes the distribution and accumulation of noise throughout the forward process. This complements the fixed-index comparisons in Table 1 and the reversed-annealing ablation in Appendix H.4, which further investigate the effect of temporal tail-index allocation.

Reviewer_W8jpOpenReview ↗
Rating
3
Confidence
1
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper extends DLPM to a time-varying stable tail index, deriving consistent conditional reverse updates and characterizing the accumulated noise. Experiments investigate whether annealing improves tail coverage and long-tailed image generation.

Critical strengths

The paper addresses a meaningful extension of heavy-tailed diffusion and provides a consistent construction rather than directly substituting a varying index into fixed-index formulas. The matched main experiments and schedule controls offer encouraging evidence of improved generation.

Critical weaknesses
  1. Schedule justification and repeatability As I go through the paper, it seems that 1.5 to 2.0 half-cosine schedule is a design choice for annealing, and the theory does not establish its optimality or sampling-error advantage. The shape, direction, and endpoint controls are useful, but each uses only one training run. Please distinguish empirical motivation from theoretical guarantees and quantify training variability for the key schedule comparisons.
  2. Theory–implementation connection Image experiments clip the stable scales at 20, while the mixed-power and polynomial-tail results concern untruncated scales. Clipping preserves conditional Gaussian updates but changes the noise tails, and removing it substantially worsens performance in Table 9. Although baselines use the same cap, please explain which theoretical insights survive clipping and provide clipping-frequency or effective-noise diagnostics to support the proposed mechanism.
Questions for authors

Regarding the first weakness, how much evidence supports the selected endpoints and annealing direction beyond the single training run per variant? Please clarify which conclusions concern local sensitivity and which support a general schedule-design principle.

4 comments

We thank the reviewer for the comments on schedule justification and on clipping in the implementation. Below, we address the reviewer's concerns and questions in detail.

C1. it seems that 1.5→2.01.5\to2.0 half-cosine schedule is a design choice for annealing, and the theory does not establish its optimality or sampling-error advantage.

The half-cosine schedule in Eq. (3) is a prespecified default rather than a theoretically established optimum. We choose it for its smoothness, monotonicity, and zero endpoint slopes, and use it unchanged across both long-tailed datasets. Theorems 1 and 2 establish the noise law and spectral properties of the annealed construction; they do not rank schedules by generation quality or sampling error.

The experiments in Appendices H.3–H.5 support the selected direction and local robustness of the default. The linear schedule and schedules with moderate curvature changes give similar mean FID., while reversing the direction substantially worsens generation quality. The evidence supports a useful shared starting default with flexibility in its shape and endpoints, rather than a unique optimum across datasets.


Q1. Regarding the first weakness, how much evidence supports the selected endpoints and annealing direction beyond the single training run per variant? Please clarify which conclusions concern local sensitivity and which support a general schedule-design principle.

Each variant below uses 3 paired training runs and 5 generation seeds per run on each dataset, and we have updated the manuscript accordingly. The standard path uses the 5 training runs and 5 generation seeds per run from the main experiment. The direction comparison provides the strongest empirical support for a broader design principle: placing near-Gaussian increments near the terminal noise and heavier-tailed increments near the data. Reversing this allocation worsens FID on both datasets. The shape and endpoint comparisons assess local sensitivity around the shared default: performance is robust to nearby starting endpoint choices, while the preferred endpoint can depend on the dataset.

These are empirical conclusions within the tested settings. Theorems 1 and 2 characterize the noise law produced by annealing, while the choice of direction, endpoints, and curvature is supported by the experiments in Appendices H.3–H.5. The default in Eq. (3) is a prespecified design, rather than a theoretically optimal schedule or one with a proved sampling-error advantage.

The table reports mean FID after averaging the 5 generation seeds within each training run.

Forward scheduleCIFAR-10-LT FIDCIFAR-100-LT FID
Half-cosine, 1.5→2.01.5\to2.017.301026.5218
Half-cosine, p=0.5p=0.5, 1.5→2.01.5\to2.017.451627.3523
Half-cosine, p=2p=2, 1.5→2.01.5\to2.017.239727.2700
Half-cosine, p=4p=4, 1.5→2.01.5\to2.017.843728.8112
Linear, 1.5→2.01.5\to2.017.381227.3672
Mirrored half-cosine, 2.0→1.52.0\to1.541.500250.7715
Half-cosine, 1.2→2.01.2\to2.018.517026.3562
Half-cosine, 1.3→2.01.3\to2.017.769425.3063
Half-cosine, 1.4→2.01.4\to2.017.372025.3866
Half-cosine, 1.6→2.01.6\to2.017.356927.9000
Half-cosine, 1.7→2.01.7\to2.017.584628.5206
Half-cosine, 1.8→2.01.8\to2.018.039330.2344
Half-cosine, 1.5→1.91.5\to1.919.529028.5913

The curvature parameter pp is defined in Appendix H.3, and p=1p=1 is the default. FID is robust to the changes p=0.5p=0.5 and p=2p=2 on both datasets.

The strongest finding is the direction comparison: reversing the whole path consistently worsens FID on both datasets. The standard path places near-Gaussian increments at the high-noise end and heavier-tailed increments near the data, consistent with the reverse-time affinity shift in Appendix F.

The linear and half-cosine means are close, supporting robustness to this change of schedule shape at fixed endpoints. Nearby starting indices retain useful performance, and lowering the starting endpoint improves CIFAR-100-LT FID. Reducing the upper endpoint worsens mean FID on both datasets. As discussed in Appendix H.5, this is consistent with placing near-Gaussian increments at the high-noise end: lowering the upper endpoint keeps the scheduled increments non-Gaussian even at the terminal time. Together, these comparisons support the selected direction across the two datasets and local flexibility in schedule shape, while the endpoint study supports treating the starting index as a choice that can be adapted to the dataset.

C2. Theory–implementation connection

Following DLPM [1] and LIM [2], we clip positive-stable mixing scales at C=20C=20 to control occasional extreme scale realizations for numerical stability during training and sampling. Clipping changes the marginal noise law while preserving the conditional Gaussian formulas, as detailed below. The same clipping threshold is applied to the fixed-index baselines, ensuring consistent comparisons. Importantly, the variance recursion and conditional Gaussian formulas in Eqs. (6), (8), and (12)–(15) remain exact after clipping, given X0X_0 and the realized scale path. Appendix H.6 and Table 9 examine the empirical effect of this implementation choice, while the diagnostics below quantify its frequency and impact on accumulated noise.

With Aj(C)=min⁡(Aj,C)A_j^{(C)}=\min(A_j,C), the residual satisfies

\sim\mathcal N\\!\left(0, \sum_j w_{j,t}^2A_j^{(C)} I_d\right).$$ Independence of the scales gives its exact characteristic exponent: $$\begin{aligned} \Psi_t^{(C)}(r) &=-\log\mathbb E\\!\left[e^{i\langle u,Y_t^{(C)}\rangle}\right]\\\\ &=-\log\mathbb E\exp\left( -\frac{r^2}{2}\sum_{j=1}^t w_{j,t}^2 A_j^{(C)} \right)\\\\ &=-\sum_{j=1}^t\log\mathbb E\exp\left( -\frac{r^2w_{j,t}^2}{2}A_j^{(C)} \right) \end{aligned}$$ for $r=\\|u\\|$. Hence the marginal residual remains a Gaussian scale mixture over the per-step scales, and clipping only caps the tail of each scale. Its moment-generating function obeys \$\\mathbb E e^{\\langle v,Y_t^{(C)}\\rangle} \\leq\\exp\\{C\\|v\\|^2\\sum_jw_{j,t}^2/2\\}\$, so the finite-step residual is sub-Gaussian. The mixed-power exponent in Theorem 1, the spectral-curvature identity in Theorem 2, and the polynomial-tail asymptotic in Proposition 9 apply to the untruncated process. Thus, for the clipped implementation, the conditional Gaussian updates remain exact, while the Laplace-transform expression above characterizes the marginal noise law in place of the untruncated mixed-power formula. Table 9 compares different finite scale caps, including the untruncated setting: | Scale cap | CIFAR-10-LT FID | CIFAR-100-LT FID | | --- | --- | --- | | 10 | 17.28 | 25.69 | | 20 | 17.24 | 27.23 | | 50 | 17.64 | 27.98 | | No cap | 24.76 | 32.41 | Each row uses one trained checkpoint and five generation seeds. Clipping contributes materially to image performance, and the finite thresholds 10, 20, and 50 all work better than no cap; the default 20 is not a narrow optimum. We also sampled 100,000 independent scale paths, using the default half-cosine schedule on a grid of 4,000 training steps shared by both datasets. The table reports clipping frequencies at selected steps and the median / 99th percentile of the cumulative conditional noise standard deviation $\sqrt{\Sigma_t}$. Before/after values use identical scale paths, with only the cap changed. | Forward index $t$ | $\alpha_t$ | $P(A_t>20)$ | Uncapped $\sqrt{\Sigma_t}$: median / p99 | Capped $\sqrt{\Sigma_t}$: median / p99 | | --- | --- | --- | --- | --- | | 1 | 1.5000 | 5.291% | 0.000608 / 0.005897 | 0.000608 / 0.002036 | | 1000 | 1.5733 | 4.190% | 0.3925 / 3.4180 | 0.2066 / 0.2197 | | 2000 | 1.7501 | 1.979% | 0.8835 / 5.8908 | 0.5635 / 0.5926 | | 3000 | 1.9269 | 0.490% | 1.2500 / 4.8858 | 1.0627 / 1.1058 | | 3999 | 2.0000 | 0% | 1.4142 / 1.4142 | 1.4142 / 1.4142 | Clipping acts most frequently in the heavier-tailed part of the schedule and substantially reduces the upper quantiles of accumulated noise. At the Gaussian endpoint, $A_t=2$ deterministically and the current increment is never clipped; the cumulative variances still retain earlier increments, with the endpoint quantiles coinciding at the displayed precision. These measurements quantify the effect of the cap alongside the conditional law above. [1] Dario Shariatian, Umut Simsekli, and Alain Durmus. Denoising Lévy Probabilistic Models. *ICLR*, 2025. https://arxiv.org/abs/2407.18609 [2] Eun Bi Yoon, Keehun Park, Sungwoong Kim, and Sungbin Lim. Score-Based Generative Models with Lévy Processes. *NeurIPS*, 36:40694–40707, 2023.
Reviewer_W8jpOpenReview ↗

Reviewer’s Follow-up

Thank you for your detailed response, which has addressed my concerns. I will consider the other reviewers’ feedback before deciding whether to raise my score.

Thank you for your encouraging feedback. We are glad that our responses have addressed your concerns.

We would also like to mention that our response to Reviewer jkLR's Q4 provides additional theoretical analysis of the clipped noise distribution, which is closely related to your second concern (Q2). We hope this analysis will be helpful in your overall assessment.

Thank you again for your time and thoughtful consideration.

Best regards

The Authors

Reviewer_xHNZOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes Annealed DLPM (A-DLPM), which replaces the fixed tail index α in the Denoising Lévy Probabilistic Model (DLPM) with a time-varying half-cosine schedule: α increases from 1.5 near the data (heavy-tailed increments for tail coverage) to 2.0 near the noise (Gaussian for easier reverse estimation). The forward process includes a signal-correction term that preserves the conditional mean of X₀; the reverse process is a conditional Gaussian whose coefficients are derived analytically. Theoretically, the residual Y_t follows a mixed-power spectrum with an exact spectral-curvature identity, and the discrete process converges in J₁-Skorokhod topology to a continuous-time ODE with an additive non-stationary process. When α is constant, A-DLPM recovers DLPM exactly. Experiments on a 2D stable-tail task and CIFAR-10-LT / CIFAR-100-LT show improvement.

Critical strengths
  1. Well-motivated core idea. The paper directly addresses the trade-off identified by Cherkaoui et al. (2026): a fixed tail index compromises between terminal initialization (from lighter tails to easier reverse) and tail coverage (from heavier tails to better rare-class generation). Allocating heavy tails near the data and light tails near the noise is a natural and principled resolution.
  2. Rigorous theoretical development. The mixed-power spectrum characterization, the exact spectral-curvature identity, the J₁-Skorokhod continuous-time limit, and the exact reduction to DLPM at constant α together form a complete theoretical package. The curvature identity in particular gives a clean mechanistic explanation: low frequencies inherit the heavy-tail behavior (rare modes), while high frequencies approach Gaussian (easy learning).
  3. Clean experimental methodology. Five paired seeds with 95% bootstrap CIs, a grid of fixed-α baselines, comparisons against DLIM/LIM/Gaussian DDPM, grouped coverage by head/medium/tail classes, and ablations on schedule shape and reversed direction all support the conclusions without obvious confounds.
  4. Backwards compatibility. When α_t is constant, the formulas reduce exactly to DLPM.
Critical weaknesses
  1. The proposed mechanism (Lévy jumps bridging modes) is not distinguished from a simpler alternative: implicit regularization/augmentation. The paper claims heavy-tailed noise helps because Lévy jumps allow reverse trajectories to traverse separated basins and reach underrepresented modes. But long-tailed imbalance is primarily a sample-frequency problem (rare classes have few training examples), not a manifold-distance problem. An alternative explanation is: heavy-tailed noise near the data acts as a stronger perturbation/augmentation during training, and rare classes, which are more prone to overfitting due to few samples, benefit more from regularization. The paper does not design an experiment to distinguish these two mechanisms. A controlled test on a balanced dataset with well-separated modes (e.g., two far-apart Gaussian clusters with equal sample counts) would discriminate: if the "mode-bridging" hypothesis is correct, heavy tails should still help; if the "regularization" hypothesis is correct, heavy tails should not help on balanced data. This experiment is absent.
  2. No balanced-data control. The paper evaluates only on long-tailed splits (CIFAR-10-LT, CIFAR-100-LT). It does not report A-DLPM's performance on balanced CIFAR-10/100. Figure 4 already hints that A-DLPM's gains are concentrated in Medium and Tail classes, with Head-class coverage barely above baselines, suggesting that heavy-tailed noise may hurt head-class reconstruction while helping tail classes. If A-DLPM degrades on balanced data, the method is a task-specific tuning rather than a general noise-law improvement.
  3. No comparison against simpler long-tailed remedies. The paper isolates the noise law by training without class balancing, auxiliary losses, or reweighting. But the practical question for a practitioner is: does changing the noise schedule outperform class-balanced sampling or standard data augmentation under the same compute? Without this comparison, the ~1.5-point FID improvement and ~2-point Recall improvement are hard to interpret, they may simply be a weaker substitute for over-sampling rare classes.
  4. The schedule hyperparameters appear tuned to CIFAR-LT. α₀ = 1.5, α_{N-1} = 2.0, half-cosine shape, these are manually selected. The paper states "nearby monotone schedules yield comparable results" but does not test whether the optimal α₀ transfers to a different imbalance ratio or a different dataset. If α₀ needs retuning per dataset, the contribution is closer to a hyperparameter search trick than a principled insight.
  5. Experimental scale is limited to CIFAR-32. No latent-space diffusion, no higher-resolution benchmarks, no modern architectures.
  6. Computational cost of stable variate sampling is unreported.
Questions for authors
  1. Endpoint sensitivity: How does performance vary as α₀ ranges from 1.2 to 1.8? Is α₀ = 1.5 optimal across datasets, or should it be tuned per dataset (e.g., based on the empirical tail index of the data)?
  2. Scale-up: Have you tried A-DLPM on latent-space diffusion (e.g., Stable-Diffusion-style VAE + DiT) or higher-resolution CIFAR? Does the half-cosine schedule transfer, or does the optimal schedule shape change?
  3. vs. data rebalancing: Under the same compute budget, how does A-DLPM (no rebalancing) compare against Gaussian DDPM with class-balanced sampling or logit adjustment? Is the noise-law change orthogonal to these methods?
  4. Computational cost: What is the per-step cost of drawing A_t ~ A_{α_t/2} versus Gaussian noise Z_t? Is the stable-variate sampling a bottleneck at 1000+ timesteps?
  5. Learned schedule: Could α_t be learned (e.g., parameterized as a small MLP and optimized end-to-end)? Is there a risk that a learned schedule overfits?
  6. Connection to Cherkaoui et al. (2026): Their decomposition of error into terminal-initialization and accumulated-training components. Does A-DLPM's schedule shift these components as predicted? Can you report the two components separately?
6 comments

We would like to thank the reviewer, who has taken great time and effort to provide in-depth comments. We address the questions and concerns below.

Q1. Endpoint sensitivity: How does performance vary as α0\alpha_0 ranges from 1.2 to 1.8? Is α0=1.5\alpha_0=1.5 optimal across datasets, or should it be tuned per dataset (e.g., based on the empirical tail index of the data)?

Our study of the starting endpoint now covers α0∈{1.2,1.3,1.4,1.5,1.6,1.7,1.8}\alpha_0\in\{1.2,1.3,1.4,1.5,1.6,1.7,1.8\} on both datasets, completing the full 1.21.2–1.81.8 sweep, and Appendices H.3–H.5 of the manuscript has been revised accordingly. α0=1.5\alpha_0=1.5 is a prespecified default used for both datasets with local robustness, but not a universal optimum.

The table reports mean FID. The standard path uses the 5 training seeds and 5 generation seeds per training run from the main experiment; each shape, mirrored-path, and endpoint variant uses 3 training seeds and 5 generation seeds per training run, with 5,000 generated images per generation against the same long-tailed reference. The table reports the mean FID over all training and generation seeds.

Forward scheduleCIFAR-10-LT FIDCIFAR-100-LT FID
Half-cosine, 1.5→2.01.5\to2.017.301026.5218
Half-cosine, p=0.5p=0.5, 1.5→2.01.5\to2.017.451627.3523
Half-cosine, p=2p=2, 1.5→2.01.5\to2.017.239727.2700
Half-cosine, p=4p=4, 1.5→2.01.5\to2.017.843728.8112
Linear, 1.5→2.01.5\to2.017.381227.3672
Mirrored half-cosine, 2.0→1.52.0\to1.541.500250.7715
Half-cosine, 1.2→2.01.2\to2.018.517026.3562
Half-cosine, 1.3→2.01.3\to2.017.769425.3063
Half-cosine, 1.4→2.01.4\to2.017.372025.3866
Half-cosine, 1.6→2.01.6\to2.017.356927.9000
Half-cosine, 1.7→2.01.7\to2.017.584628.5206
Half-cosine, 1.8→2.01.8\to2.018.039330.2344
Half-cosine, 1.5→1.91.5\to1.919.529028.5913

Half-cosine and linear schedules give close means on both datasets. Reversing the entire path produces large FID increases, supporting the selected orientation across training runs. Lowering the starting endpoint improves CIFAR-100-LT FID relative to the default, so the endpoint α0=1.5\alpha_0=1.5 is a prespecified default rather than a post-hoc universal optimum. The tested nearby starting indices retain comparable performance, supporting local robustness around the shared default.

Here pp is the power of the half-cosine schedule defined in Appendix H.3, with p=1p=1 as the default. Paired FID intervals for p=0.5p=0.5 and p=2p=2 remain close to the standard path on both datasets. These results support local flexibility in the schedule shape.

Class imbalance specifies frequencies, whereas the stable index specifies a perturbation law. Relating them requires a model connecting those quantities, which remains a question for future research.

Q2. Scale-up: Have you tried A-DLPM on latent-space diffusion (e.g., Stable-Diffusion-style VAE + DiT) or higher-resolution CIFAR? Does the half-cosine schedule transfer, or does the optimal schedule shape change?

Due to limited computational resources, we leave higher-resolution experiments, latent diffusion, and DiT evaluation to future research. The current image evidence is for the 32-by-32 improved-DDPM U-Net in Section 5.2 and Appendix G.2. Indeed, the DLPM and LIM papers that we follow also evaluate at 32×32 CIFAR-10, and our comparisons follow their protocol. The construction in Section 3 and the reverse formulas in Appendix B.2 apply in arbitrary finite dimension and accommodate different denoising architectures. Thus the mathematical construction extends to these settings.


Q3. vs. data rebalancing: Under the same compute budget, how does A-DLPM (no rebalancing) compare against Gaussian DDPM with class-balanced sampling or logit adjustment? Is the noise-law change orthogonal to these methods?

We evaluate tail-index annealing and class-balanced sampling under the same training budget, including their joint use. This comparison examines two interventions: changing the perturbation distribution and changing the frequencies with which training examples are selected.

We use the same U-Net, budget of training updates, and augmentation by horizontal flips, with three paired training seeds and one shared generation seed. Rebalancing samples images from the original long-tailed pool with sampling weights inversely proportional to class frequency. Labels are used only by the sampler. All denoisers remain unconditional. Every entry uses 5,000 generated images and the same class-balanced reference of 5,000 images from the original training set.

DatasetTraining settingFID ↓Recall ↑Coverage ↑
CIFAR-10-LTNatural A-DLPM32.09870.62080.6125
CIFAR-10-LTRebalanced DDPM28.84050.59010.7004
CIFAR-10-LTRebalanced A-DLPM28.89540.53940.6759
CIFAR-100-LTNatural A-DLPM28.75450.57130.7515
CIFAR-100-LTRebalanced DDPM33.11070.56570.7269
CIFAR-100-LTRebalanced A-DLPM32.34690.52970.7192

Class-balanced sampling changes the distribution seen by the denoiser, bringing class frequencies closer to the balanced evaluation reference. At a fixed training budget, it increases repeated exposure to the limited rare-class images and reduces the relative exposure to common-class examples. This creates a trade-off between matching class frequencies and learning within-class variation. The different results on CIFAR-10-LT and CIFAR-100-LT are consistent with this trade-off, particularly given the smaller number of examples per class in CIFAR-100-LT.

Tail-index annealing acts on the perturbation law, and class-balanced sampling acts on the selection of training examples. The two interventions can be implemented independently or jointly, as demonstrated by the rebalanced A-DLPM experiments. Their performance effects interact through the denoising objective, and the combined results show that changes in FID, Recall, and Coverage can differ. The CIFAR-100-LT comparison retains a mean FID advantage for A-DLPM under class-balanced sampling.

Evaluating logit adjustment would require an objective with class logits, while our unconditional denoiser predicts noise.

Q4. Computational cost: What is the per-step cost of drawing At∼Aαt/2A_t\sim\mathcal A_{\alpha_t/2} versus Gaussian noise ZtZ_t? Is the stable-variate sampling a bottleneck at 1000+ timesteps?

In our benchmark of the current implementation, preparing the scale path accounts for a small fraction of the measured training computation. The Chambers–Mallows–Stuck (CMS) algorithm [1] generates a positive-stable scale directly. Let p=αt/2∈(0,1)p=\alpha_t/2\in(0,1), U∼Uniform(0,π)U\sim\mathrm{Uniform}(0,\pi), and E∼Exp(1)E\sim\mathrm{Exp}(1) be independent. Under our scale convention,

\left(\frac{\sin((1-p)U)}{E}\right)^{(1-p)/p},$$ with $A_t=2$ when $\alpha_t=2$. Thus, each scale requires only two elementary random draws and a fixed number of arithmetic and transcendental operations, giving the same $O(1)$ per-sample complexity as Gaussian sampling via the Box–Muller transform. Computing a training marginal requires $O(N)$ scalar scale draws and variance accumulation per example, with each scale shared across all image coordinates. During generation, the scale path and cumulative variances are prepared once before the reverse loop, without additional network evaluations. We measured the current implementation on an RTX 4090 , using batch size 100, the paper's U-Net for 32×32 images and a training grid of 4,000 steps, five warm-up calls, and 20 synchronized wall-clock measurements per operation. The benchmark uses synthetic inputs and a randomly initialized network. The training computation includes the actual loss, backward pass, gradient clipping, and AdamW update, excluding data loading and EMA updates. | Operation | Mean time (ms) | | --- | --- | | Gaussian image noise, $100\times3\times32\times32$ | 3.54 | | Scale sampling and variance accumulation, measured separately | 6.43 | | Scale sampling and variance accumulation, measured within training | 7.62 | | Training computation, including scale preparation within the loop | 401.99 | Preparing the scale path accounts for 1.90% of the measured training computation, using the timings recorded within the same training iterations. At 1,000 or more steps, the scale path is still prepared once before the reverse loop, so the per-step cost does not grow with the number of timesteps and stable-variate sampling is not a bottleneck. [1] John M. Chambers, Colin L. Mallows, and B. W. Stuck. A Method for Simulating Stable Random Variables. *Journal of the American Statistical Association*, 71(354):340–344, 1976. https://doi.org/10.1080/01621459.1976.10480344

Comments on the paper

1 comment
AuthorsOpenReview ↗

Summary of the Contribution and Author Discussion

Dear Area Chairs, Senior Area Chairs, Program Chairs, and Reviewers,

We would like to provide a brief summary of our paper's main contributions, experimental findings, and the additional analyses conducted during the discussion.

Our paper asks whether a diffusion model should use the same tail index near clean data as it does near terminal noise. A-DLPM allows the index to vary across time. This raises a mathematical question: once increments with different stable indices are added, what distribution does the denoiser encounter, and how can the corresponding reverse updates be derived? We derive the forward marginal and reverse updates between arbitrary retained times by conditioning on the scale path and clean endpoint. For the untruncated process, we characterize how different indices contribute to the spectrum of the accumulated noise and derive its continuous-time limit. At a constant index, the construction recovers DLPM[1] exactly.

Under our matched training protocol, A-DLPM achieves lower mean FID and higher mean Recall than Gaussian DDPM, all evaluated DLPM and DLIM variants[1], and most LIM variants[2] on both long-tailed benchmarks. It achieves the highest mean Recall among all evaluated methods. Its FID advantage persists when evaluation uses a class-balanced reference. The strongest result of the schedule ablations is the large sensitivity to annealing direction, compared with the smaller changes observed when varying the starting index. Reversing the half-cosine schedule substantially worsens FID on both CIFAR-10-LT and CIFAR-100-LT. Appendix F provides complementary evidence: early in reverse sampling, A-DLPM's normalized prediction-error profile resembles that of DLPM with Gaussian noise (α=2.0\alpha = 2.0), while closer to the data, it resembles that of DLPM with heavier-tailed noise (α=1.5\alpha = 1.5). In response to Reviewer xHNZ's suggestion, we tested two equally weighted Gaussian components at different separations. Compared with Gaussian DDPM, A-DLPM achieved lower mean sliced Wasserstein distances and smaller mean errors in generated mode proportions at both separations. On balanced CIFAR-10 and CIFAR-100, A-DLPM achieved comparable FID to Gaussian DDPM, while higher starting indices improved A-DLPM's mean FID.

Several reviewers raised questions about clipping and its theoretical implications (W8jp, C2; 5fZ1, Q4; jkLR, Q4). The conditional Gaussian formulas remain exact with clipped scales, and we derived the corresponding marginal characteristic exponent and tail bounds. In response to jkLR’s Q5, we evaluated a Gaussian control that matches A-DLPM’s signal attenuation and the variance and cross-time covariance of the clipped noise at C=20C=20. The control achieved comparable FID and Recall on both datasets. Together with the schedule ablations, this suggests that the temporal allocation of noise induced by annealing plays an important role in generation quality. We also derived a TV bound for marginal distributions at positive times, addressing jkLR Q6. Under the stated integrability assumption, the bound has one term for terminal-initialization error and another for accumulated endpoint-prediction errors weighted by the sensitivity of each reverse update.

We thank the reviewers for their constructive feedback and the chairs for their time and consideration.

Best regards,

The Authors

[1] Dario Shariatian, Umut Simsekli, and Alain Durmus. Denoising Lévy Probabilistic Models. ICLR, 2025. https://arxiv.org/abs/2407.18609

[2] Eun Bi Yoon, Keehun Park, Sungwoong Kim, and Sungbin Lim. Score-Based Generative Models with Lévy Processes. NeurIPS, 36:40694–40707, 2023.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.