acceptodds
Under review as a conference paper at ICLR 2027

-Power Heavy-Tailed Diffusion Models

Abstract

We develop a variational framework for heavy-tailed diffusion based on an (\ell_p )-power family, extending Student- (t )-based models beyond (p=2 ). Using the family’s Gaussian scale-mixture structure, we construct diffusion paths for (p\in[1,2] ) via random-precision Gaussians and derive an adjacent-state -th power objective through (\gamma )-power-divergence projection. At (p=2 ), the objectives are equivalent up to a constant, but differ for (p\in[1,2) ). Theoretical analysis identifies regimes where the adjacent-state objective is more sensitive to rare components than denoising, and conditions under which (p\in(1,2\) ) yields lower reconstruction risk than either endpoint. Experiments on synthetic data, language, and image generation reveal complementary strengths over Gaussian and Student- (t ) baselines: the adjacent-state objective better preserves rare modes under heavy-tailed noise, while denoising achieves better reconstruction on contaminated data.

Primary area:
probabilistic methods (Bayesian methods, variational inference, sampling, UQ, etc.)
Keywords:
Heavy-tailed diffusion models, $\ell_p$-power distributions, $\gamma$-power divergence, Gaussian scale mixtures, Image denoising

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_S3HdOpenReview ↗
Rating
2
Confidence
2
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

The paper develops a variational framework for heavy-tailed generative modeling using an ℓp\ell_p-power family that includes Student-t and symmetric Pareto distributions. It characterizes Gaussian scale-mixture representations and constructs temporally coupled diffusion paths. For fixed decoder scales, joint γ\gamma-power-divergence projection yields an adjacent-state ℓp\ell_p-power prediction loss. The paper shows that replacing the adjacent target with the clean-data target preserves the objective up to an additive constant at p=2p = 2, but generally changes the objective and its population minimizer below p=2p = 2.

The analysis relates these minimizers to conditional Fréchet pp-centers and identifies conditions governing rare-component sensitivity and reconstruction risk. Experiments include imbalanced heavy-tailed mixtures, character-level typo correction on WikiText-103, a controlled colored-MNIST estimator study, and image restoration and generation from contaminated training sets. The results support a distinction between preserving valid rare structure and suppressing corrupted targets. The exact variational construction concerns a discrete stochastic model; the deterministic sampler used in the main experiments is explicitly acknowledged as heuristic.

Critical strengths
  1. A consequential distinction between training targets. Theorem 3.4 derives an adjacent-state objective from a joint path projection without assuming a KL-style additive chain rule. Theorem 3.6 then establishes the special role of the quadratic case and the general failure of objective equivalence below it. Proposition 3.7 provides a mechanism for different responses to rare conditional components, while Corollary 3.8 identifies a condition under which greater sensitivity increases reconstruction risk. These results make the contribution more substantial than introducing an intermediate loss exponent. The family and scale-mixture ingredients build on established work; the valuable contribution is their integration into a path objective and the analysis of the prediction targets.

  2. Substantial controlled validation with explicit boundaries. Appendices F.2 and F.3 use matched architectures and training conditions and report results over three seeds. The noise–loss ablations in Tables 6–7 and the dimension/noise sweep in Table 3 directly probe the proposed explanation. At p=1.2p = 1.2, adjacent training reduces the mixture discrepancy from 0.062 to 0.012 relative to matched denoising training (Table 2). The image results show substantial improvements under the specified contaminated-training regime; for example, CelebA FID is 37.176 for p=1.2p = 1.2 denoising versus 267.229 for DDPM (Table 5). Appendix G also acknowledges conflicting fidelity–coverage rankings and the limits of the theoretical explanation. This combination constitutes a meaningful body of work.

Critical weaknesses
  1. The exact model and the empirically evaluated sampler are not sufficiently connected. Theorem 3.4 concerns the coupled discrete reverse model, whose exact stochastic sampler maintains an accumulated residual statistic. However, the mixture and image experiments use the deterministic ODE branch. Appendices E.3 and G correctly acknowledge that a conditional pp-center generally does not identify a conditional-mean probability-flow velocity, so marginal preservation does not follow from the discrete projection. The acknowledgement is valuable, but the paper does not empirically establish how results change when using the exact stochastic model, the ODE heuristic, or direct prediction. This limits the extent to which the application results validate the variational construction, while leaving the discrete theoretical contribution intact.

  2. Application-level gains are less well isolated than the synthetic objective comparison. Table 7 already shows that Student-t noise combined with the same nonquadratic penalty can perform similarly to the matched ℓp\ell_p-power family: at p=1.2p = 1.2, their adjacent-training discrepancies are 0.013 and 0.012. Yet the image experiments do not include the corresponding Gaussian/Student-t controls with the same penalty, prediction parameterization, and sampling procedure. Appendix F.5 also changes prediction parameterization and training weighting across methods. Consequently, the large image gains cannot yet be cleanly attributed to the new noise family rather than robust loss design and other choices. Robust diffusion objectives are acknowledged in Appendix A but are not directly evaluated in this setting.

    The language experiment presents another attribution issue: at p=1.2p = 1.2, head-word recall falls from 0.991 under adjacent training to 0.103 under denoising training (Table 4). This broad degradation exceeds a selective loss of rare words. Appendix G acknowledges possible effects of learned embeddings, auxiliary classification, and optimization, but leaves them unresolved. These limitations reduce the confidence with which the application differences can be attributed to the population-level mechanism.

Questions for authors
  1. Can you compare the exact stochastic branch of Algorithm 1, the deterministic ODE branch, and direct prediction using the same trained predictors on a representative setting? Does the adjacent-versus-denoising ordering survive the sampler choice? This would materially affect the connection between the variational construction and the reported practical behavior.

  2. Do the restoration and generation gains remain when Gaussian or Student-t perturbations use the same nonquadratic loss, prediction parameterization, and comparable sampler? The synthetic controls in Table 7 are informative; the corresponding application-level control would clarify the added practical value of the ℓp\ell_p-power family. A comparison with an appropriate robust diffusion loss would also help establish this distinction.

  3. What explains the severe head-word degradation of the p=1.2p = 1.2 denoising model in Table 4? Can the existing training diagnostics or a control with fixed embeddings and matched auxiliary-loss weighting distinguish a target-induced statistical effect from an optimization or representation-learning failure? The answer could change how strongly I regard this experiment as evidence for the proposed mechanism.

  4. Please specify the generated and reference image counts and feature-extraction protocol used for FID and precision/recall, and explain how the selected configurations in Table 9 were chosen. Appendix F.5 describes training in detail but does not make these evaluation choices equally explicit. This is a reproducibility clarification, not a request for additional application domains.

4 comments

Thank you for the careful and technically insightful review. Below, we provide additional experiments, explain their implications and limitations, and clarify the evaluation protocol.

Response to Weakness 1 & Question 1

We evaluated the sampling procedures on CIFAR-10 with 75% contaminated training targets. For each ℓ1.2\ell_{1.2} model and seed, all samplers use the same EMA checkpoint obtained after 30k training updates, with x0x_0-prediction and no retraining between samplers. “Joint” denotes the exact stochastic branch of Algorithm 1 for the specified discrete reverse model, which maintains the accumulated residual statistic. We compare this branch with the deterministic Heun branch and direct prediction using a single network evaluation, and additionally report ancestral sampling.

For context, we also include DDPM and t-EDM baselines. Their regular runs use 50k training updates; the extended-budget runs, marked with *, use 200k updates for DDPM and 100k for t-EDM. NFE is reported per sampling trajectory.

Table C1. Sampling comparison on CIFAR-10 with 75% contaminated training targets.

ModelPredictionSamplerNFEPSNR ↑FID ↓Precision ↑Recall ↑
p=1.2p=1.2, Dns.x0x_0Heun1936.600 ± 0.24785.828 ± 4.7480.374 ± 0.0100.463 ± 0.003
p=1.2p=1.2, Dns.x0x_0Joint2033.059 ± 0.097220.463 ± 7.7130.133 ± 0.0040.007 ± 0.004
p=1.2p=1.2, Dns.x0x_0Direct134.146 ± 0.332362.135 ± 13.3950.461 ± 0.1730.000 ± 0.000
p=1.2p=1.2, Dns.x0x_0Ancestral2036.228 ± 0.895221.766 ± 4.5710.161 ± 0.0050.340 ± 0.014
p=1.2p=1.2, Adj.x0x_0Heun1935.204 ± 0.441302.162 ± 22.4690.059 ± 0.0210.431 ± 0.042
p=1.2p=1.2, Adj.x0x_0Joint2032.901 ± 0.151304.952 ± 16.6430.029 ± 0.0120.010 ± 0.009
p=1.2p=1.2, Adj.x0x_0Direct125.861 ± 2.977437.875 ± 5.6800.007 ± 0.0040.000 ± 0.000
p=1.2p=1.2, Adj.x0x_0Ancestral2031.630 ± 0.647391.604 ± 2.2970.000 ± 0.0000.611 ± 0.071
DDPMϵ\epsilonAncestral2037.119 ± 0.030190.550 ± 8.6670.209 ± 0.0110.415 ± 0.006
DDPMϵ\epsilonHeun3931.189 ± 1.093137.574 ± 32.7180.291 ± 0.0580.518 ± 0.002
DDPMϵ\epsilonDirect17.62 ± 0.16361.8 ± 9.60.007 ± 0.0040.000 ± 0.000
DDPM*ϵ\epsilonAncestral100036.777 ± 0.205268.460 ± 5.9840.103 ± 0.0070.480 ± 0.004
t-EDMEDMHeun3935.866 ± 0.420261.056 ± 8.1190.112 ± 0.0060.403 ± 0.018
t-EDM*EDMHeun3534.809 ± 0.266288.182 ± 0.5190.086 ± 0.0020.433 ± 0.039

Dns. achieves higher mean PSNR and lower mean FID than Adj. across all four samplers, although absolute performance varies substantially. Heun–Dns. achieves the lowest mean FID, while Joint and direct sampling show limited coverage. Ancestral Adj. has high recall but very low precision, highlighting the need to consider both metrics.

Response to Weakness 2 & Question 2

We added Gaussian and Student-tt controls on CIFAR-10 with 75% contaminated training targets. All configurations in Table C2 use x0x_0-prediction, the same loss weighting, 30k training updates, and Heun sampling with 19 NFE. We vary the noise family and residual penalty. Here, ℓ1.21.2\ell_{1.2}^{1.2} and ℓ22\ell_2^2 denote the nonquadratic and squared residual penalties, respectively. The DDPM and t-EDM baseline configurations are reported separately in Table C1.

Table C2. Noise–loss controls with common prediction parameterization, loss weighting, training budget, and sampling procedure.

Noise familyResidual penaltyPSNR ↑FID ↓Precision ↑Recall ↑
ℓ1.2\ell_{1.2}-powerℓ1.21.2\ell_{1.2}^{1.2}36.600 ± 0.24785.828 ± 4.7480.374 ± 0.0100.463 ± 0.003
Student-ttℓ1.21.2\ell_{1.2}^{1.2}36.479 ± 0.21069.149 ± 2.3420.398 ± 0.0040.443 ± 0.002
Gaussianℓ1.21.2\ell_{1.2}^{1.2}36.948 ± 0.449111.944 ± 15.1800.302 ± 0.0210.500 ± 0.003
ℓ1.2\ell_{1.2}-powerℓ22\ell_2^235.405 ± 0.234324.933 ± 2.0440.046 ± 0.0010.378 ± 0.076
Student-ttℓ22\ell_2^233.830 ± 0.628317.523 ± 5.1390.057 ± 0.0010.393 ± 0.020

Replacing squared loss with the nonquadratic penalty improves all four reported mean metrics for both heavy-tailed noise families. With the penalty fixed, Student-tt achieves the lowest mean FID and highest mean precision, while Gaussian noise achieves the highest mean PSNR and recall. The ℓ1.2\ell_{1.2}-power model lies between them.

These controls highlight the contribution of nonquadratic losses and show that the gains cannot be attributed solely to the proposed noise family. Our framework connects the distributional construction to the training objective through matched divergence projection.

Response to Question 3

We find that the previously reported head-word recall collapse of p=1.2p=1.2 Dns. is resolved in the revised experiments using 32-bit floating-point precision and gradient clipping with a threshold of 20, although a substantial performance gap relative to Adj. remains on other metrics.

These changes were introduced to improve numerical stability. Gradient clipping is intended to control the large gradients that can arise under heavy-tailed perturbations. For p=1.2p=1.2 Dns., head-word recall increases from 0.103 to 0.993, while F0.5F_{0.5} increases from 0.068 to 0.604. These results indicate that the original collapse was sensitive to the numerical and optimization settings and should not be interpreted as direct evidence of the population-level mechanism.

Table C3. Updated language results using 32-bit floating-point precision and gradient clipping with a threshold of 20.

ModelF0.5F_{0.5} ↑Accuracy ↑Sentence accuracy ↑Head recall ↑Tail recall ↑
Norvig-100.6650.6140.1170.4830.113
p=1.2p=1.2, Adj.0.868 ± 0.0080.891 ± 0.0060.398 ± 0.0090.998 ± 0.0010.445 ± 0.009
p=1.2p=1.2, Dns.0.604 ± 0.0110.700 ± 0.0060.050 ± 0.0040.993 ± 0.0020.112 ± 0.006
p=1.8p=1.8, Adj.0.815 ± 0.0080.857 ± 0.0080.313 ± 0.0080.989 ± 0.0010.363 ± 0.008
p=1.8p=1.8, Dns.0.500 ± 0.0100.684 ± 0.0090.033 ± 0.0030.939 ± 0.0040.085 ± 0.006
t-EDM0.721 ± 0.0040.735 ± 0.0020.185 ± 0.0050.988 ± 0.0020.182 ± 0.003
DDPM0.722 ± 0.0030.754 ± 0.0060.255 ± 0.0030.971 ± 0.0060.250 ± 0.005

At p=1.2p=1.2, both methods achieve high head-word recall (0.993 for Dns. and 0.998 for Adj.), but Adj. retains substantially higher tail-word recall (0.445 versus 0.112), F0.5F_{0.5}, overall accuracy, and sentence accuracy. The Adj.–Dns. ordering also persists at p=1.8p=1.8.

The updated results support an empirical difference between the training targets under the revised numerical settings. However, they do not fully isolate the population-level mechanism from optimization and representation-learning effects. We will update Table 4 in the revised manuscript and the experimental description to report the use of 32-bit precision and gradient clipping.

Response to Question 4

We use a common evaluation protocol across datasets:

  • FID: We compare 50,000 generated images with 50,000 clean reference images using Inception-V3 pool3 features with TensorFlow FID weights.
  • Precision and recall: We use the neighborhood-based estimator of Kynkäänniemi et al. (2019) with k=3k=3, using the same Inception features and the same sets of 50,000 generated and 50,000 reference images.
  • PSNR: We evaluate the test set of each dataset, including all 10,000 test images for CIFAR-10. For each test image, we use one corrupted observation, average K=16K=16 restoration outputs, and clip the average to the valid image range before computing PSNR against the clean target.

The configurations in Table 9 were selected based on the PSNR–FID tradeoff, jointly considering reconstruction accuracy and distributional quality. We will explicitly document this selection criterion and the evaluation protocol in Appendix F.5.

We hope that these additional experiments and clarifications address the reviewer’s questions.

Thank you for the additional experiments and clarifications. The sampler comparisons, matched noise–loss controls, and numerical-stability checks address several of my original concerns. I also appreciate the clarification of the evaluation protocol.

The theoretical distinction between adjacent-state and denoising objectives remains interesting. However, the new controls do not establish a clear practical advantage for the proposed noise family, and the strongest application results still rely on the heuristic sampler. The connection between the variational construction and the demonstrated practical gains therefore remains limited.

Overall, I appreciate the authors’ efforts during the discussion and maintain my original rating of weak rejection.

Thank you for the follow-up. We would like to clarify the reasons for choosing the ℓp\ell_p-power family and the role of sampler selection in our practical results.

1. Theoretical role of the proposed noise family

The proposed ℓp\ell_p-power family provides a consistent connection between heavy-tailed paths, γ\gamma-power divergence, and the adjacent-state ℓpp\ell_p^p objective. Although the theoretically matched noise–loss combination need not achieve the best empirical performance in every setting, it provides a principled basis for deriving Adj. and analyzing how replacing its target with a conditional mean changes the optimal predictor under non-quadratic losses.

2. Exact stochastic sampling in the language experiments

Regarding sampling, we acknowledge that practical performance depends on the task and sampling procedure. Importantly, the language experiments use the exact stochastic Joint sampler, a detail omitted from the original experimental description. Under the updated numerical settings in Table C3, Adj. with p=1.2p=1.2 achieves an F0.5F_{0.5} score of 0.868 and tail-word recall of 0.445, compared with 0.722 and 0.250 for DDPM Thus, meaningful application gains are also obtained using the discrete reverse model specified in our construction.

3. Task-dependent sampler selection

For image generation, we acknowledge that our strongest results rely on the heuristic Heun sampler. However, this choice concerns inference and does not alter our training framework. A key practical consideration is whether stochastic perturbations during sampling are beneficial for the task. Just as Dns. is adopted as a surrogate training objective, sampler selection can also be viewed as a task-specific decision at inference. Taken together, these findings can be interpreted in terms of task-dependent sampler selection and the practical role of randomness.

Reviewer_Un68OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper proposes an ℓp\ell_p-power variational framework for heavy-tailed diffusion models, extending Student-t diffusion to 1≤p≤21\leq p\leq2. Using Gaussian scale-mixture representations and γ\gamma-power divergence, it derives new training objectives. Experiments on synthetic mixtures, typo correction, and image restoration and generation are conducted to evaluate the proposed framework.

Critical strengths
  1. This paper provides a novel variational framework for heavy-tailed diffusion models.
  2. The paper is easy to read and follow.
  3. The experiments show promising task-specific improvements over DDPM, particularly in typo correction and learning from contaminated images.
Critical weaknesses

In my opinion, given the effectiveness of existing Gaussian diffusion models, a new variational framework should demonstrate a clear empirical or theoretical advantage. The paper’s advantages in these respects remain insufficiently established.

  1. Empirically, the improvements over DDPM remain limited in scope. The image denoising and generation experiments use training sets with 75% contaminated targets, leaving performance under standard clean data training unclear. In the synthetic rare-mode experiment, the proposed method achieves rare-mode recall comparable to DDPM. A broader, more comprehensive evaluation is needed to determine when the proposed framework offers a meaningful advantage.

  2. Theoretically, Section 3 provides a variational derivation and comparisons between training objectives under idealized assumptions. However, it remains unclear whether the new framework can lead to improved training efficiency, distributional accuracy, or statistical efficiency relative to DDPM. Therefore, their significance for generative modeling remains unclear.

Questions for authors
  1. Have the authors evaluated standard unconditional generation on real-world heavy-tailed datasets, and compared against DDPM under comparable training and sampling budgets?
  2. Can the authors provide theoretical evidence on the advantage of the proposed framework?
3 comments

Thank you for your thoughtful comments on the empirical scope and theoretical significance of our framework. Following your suggestions, we conducted additional experiments with three main findings:

  • Clean generation: The p=1.8p=1.8 models achieve the lowest mean FID among the reported configurations on clean CIFAR-10.
  • Contamination robustness: p=1.2p=1.2 Dns. achieves the lowest mean FID at every tested nonzero contamination level, indicating that the preferred exponent depends on the data condition.
  • Rare-class coverage: On CIFAR-10-LT and CIFAR-100-LT, p=1.2p=1.2 Adj. achieves the lowest mean FID and highest tail recall among the reported configurations. All additional results are reported as mean ±\pm standard deviation over three seeds. The ℓp\ell_p-power models use 30k training iterations, while DDPM and t-EDM use 50k.

Response to Weakness 1 & Question 1

1. Clean unconditional generation on CIFAR-10.

We first evaluate standard unconditional generation on clean CIFAR-10, without the restoration task.

ModelFID ↓Precision ↑Recall ↑
p=1.2p=1.2, Dns.11.137 ± 0.3600.554 ± 0.0110.551 ± 0.003
p=1.2p=1.2, Adj.8.737 ± 0.0200.566 ± 0.0040.574 ± 0.001
p=1.8p=1.8, Dns.8.242 ± 0.0510.583 ± 0.0010.569 ± 0.008
p=1.8p=1.8, Adj.8.236 ± 0.1710.582 ± 0.0000.574 ± 0.005
t-EDM9.351 ± 0.1330.569 ± 0.0020.590 ± 0.001
DDPM (ϵ\epsilon-pred)33.707 ± 4.6580.503 ± 0.0050.586 ± 0.001
DDPM (x0x_0-pred)8.991 ± 0.0300.538 ± 0.0000.547 ± 0.001

All models use Heun sampling, with 19 network function evaluations (NFE) for the proposed models and 39 for the other configurations. Our models predict x0x_0, and t-EDM uses EDM preconditioning.

The p=1.8p=1.8 models achieve the two lowest mean FID scores, both near 8.24, whereas t-EDM achieves the highest mean recall. These results demonstrate favorable clean-generation performance under the reported configurations, with differences in training budgets, sampling budgets, and prediction parameterizations stated explicitly.

2. Restoration and generation across contamination levels.

We next evaluate models trained for both restoration and generation, varying the fraction of contaminated training targets over ρ∈{0,0.25,0.5,0.75}\rho\in\{0,0.25,0.5,0.75\}.

PSNR ↑ across contamination levels

Modelρ=0\rho=0ρ=0.25\rho=0.25ρ=0.5\rho=0.5ρ=0.75\rho=0.75
p=1.2p=1.2, Dns.38.334 ± 0.03937.091 ± 0.13636.804 ± 0.03536.600 ± 0.247
p=1.8p=1.8, Dns.38.379 ± 0.05235.957 ± 0.53035.341 ± 2.09435.498 ± 0.911
t-EDM38.621 ± 0.05936.715 ± 0.18136.936 ± 0.17635.866 ± 0.420
DDPM (ϵ\epsilon-pred)37.956 ± 0.06835.155 ± 0.30729.279 ± 3.06331.189 ± 1.093
DDPM (x0x_0-pred)38.407 ± 0.02136.709 ± 0.38236.234 ± 0.14736.023 ± 0.270

FID ↓ across contamination levels

Modelρ=0\rho=0ρ=0.25\rho=0.25ρ=0.5\rho=0.5ρ=0.75\rho=0.75
p=1.2p=1.2, Dns.13.840 ± 0.25331.025 ± 2.89650.843 ± 0.02385.828 ± 4.748
p=1.8p=1.8, Dns.11.514 ± 0.29999.912 ± 2.452170.078 ± 18.052266.586 ± 11.748
t-EDM14.393 ± 0.27963.746 ± 3.817151.792 ± 2.832261.056 ± 8.119
DDPM (ϵ\epsilon-pred)28.786 ± 1.96954.040 ± 0.106120.758 ± 10.451137.574 ± 32.718
DDPM (x0x_0-pred)11.910 ± 0.14282.041 ± 10.100191.605 ± 0.463294.025 ± 4.244

p=1.2p=1.2 Dns. achieves the lowest mean FID at every tested nonzero contamination level, whereas p=1.8p=1.8 Dns. achieves the lowest mean FID on clean data. PSNR rankings vary across contamination levels. Thus, the FID advantage under contamination extends beyond the original ρ=0.75\rho=0.75 setting, while the preferred exponent depends on the data condition.

Response to Weakness 1 & Question 1 (cont'd)

3. Unconditional generation on class-imbalanced image datasets.

We agree that the original synthetic experiment alone did not establish higher rare-mode recall than DDPM. We therefore evaluate unconditional generation on CIFAR-10-LT and CIFAR-100-LT.

For CIFAR-10-LT, the tail comprises three classes with 139, 83, and 50 training images. For CIFAR-100-LT, it comprises 33 classes with at most 22 images per class, including a rarest class with only five images. Evaluation uses balanced clean references. Tail-R measures the fraction of real tail-class images covered by the generated feature manifold; NFE counts network evaluations per sampling trajectory.

CIFAR-10-LT

ModelSamplerNFETraining updatesFID ↓Precision ↑Recall ↑Tail-R ↑
p=1.2p=1.2, Dns.Heun1930k24.092 ± 0.0300.566 ± 0.0040.524 ± 0.0090.526 ± 0.001
p=1.2p=1.2, Adj.Heun1930k23.858 ± 0.1950.578 ± 0.0030.534 ± 0.0050.536 ± 0.002
p=1.8p=1.8, Dns.Heun1930k25.100 ± 0.4740.610 ± 0.0080.490 ± 0.0080.468 ± 0.003
p=1.8p=1.8, Adj.Heun1930k24.771 ± 0.1830.602 ± 0.0010.496 ± 0.0110.472 ± 0.028
t-EDMHeun3950k26.506 ± 0.4990.607 ± 0.0020.484 ± 0.0030.465 ± 0.015
DDPMHeun3950k44.330 ± 1.3620.560 ± 0.0140.460 ± 0.0050.456 ± 0.027
DDPMAncestral2050k52.321 ± 5.2590.596 ± 0.0180.312 ± 0.0040.304 ± 0.005

CIFAR-100-LT

ModelSamplerNFETraining updatesFID ↓Precision ↑Recall ↑Tail-R ↑
p=1.2p=1.2, Dns.Heun1930k23.136 ± 0.2320.570 ± 0.0050.459 ± 0.0020.415 ± 0.003
p=1.2p=1.2, Adj.Heun1930k21.947 ± 0.1330.580 ± 0.0040.465 ± 0.0010.422 ± 0.010
p=1.8p=1.8, Dns.Heun1930k24.657 ± 0.1130.625 ± 0.0020.417 ± 0.0060.371 ± 0.006
p=1.8p=1.8, Adj.Heun1930k24.555 ± 0.0540.618 ± 0.0000.431 ± 0.0050.385 ± 0.003
t-EDMHeun3950k25.898 ± 0.1910.607 ± 0.0050.406 ± 0.0040.356 ± 0.008
DDPMHeun3950k39.922 ± 0.1060.575 ± 0.0100.365 ± 0.0040.317 ± 0.001
DDPMAncestral2050k52.936 ± 0.6260.587 ± 0.0010.228 ± 0.0030.196 ± 0.002

On both datasets, p=1.2p=1.2 Adj. achieves the lowest mean FID and highest overall and tail recall among the reported configurations, using fewer training updates and NFE than the baselines. Meanwhile, p=1.8p=1.8 Dns. achieves the highest mean precision, illustrating a fidelity–coverage tradeoff. These experiments directly assess rare-class coverage in real-image datasets with long-tailed class distributions.

Response to Weakness 2 & Question 2:

Thank you for asking us to clarify the theoretical motivation and supporting evidence for our framework.

From a divergence-projection perspective, our motivation is to reduce residual-distribution misspecification when Gaussian assumptions are inappropriate. As summarized in Table 1 of our general response, our framework considers the ℓp\ell_p-power family with 1≤p≤21\leq p\leq2 and ν>p\nu>p, recovering the Gaussian family as ν→∞\nu\to\infty at p=2p=2. These parameters control distinct aspects of the model: ν\nu governs tail decay, whereas pp governs residual geometry and the associated prediction criterion.

For ν\nu, prior work provides empirical motivation for allowing finite degrees of freedom. Kim et al. [1] formulate Student-tt VAEs through joint γ\gamma-power divergence minimization and report improved modeling of low-density regions in heavy-tailed data. Extending Student-tt modeling to diffusion, Pandey et al. [2] introduce t-EDM and report improved tail estimation on weather data. In their unconditional HRRR experiments, EDM and t-EDM use the same reported sampling budget of 18 NFE (Table 5). Together, these findings support the practical value of flexible tail modeling and motivate retaining ν\nu as a tunable parameter in our variational framework.

Building on this empirical motivation, we additionally examine how pp controls sensitivity to prediction residuals. We have added this discussion to the Introduction of the revised manuscript. Heavy-tailed noise or targets can produce large residuals that dominate stochastic gradients under squared loss. For a scalar residual rr, the derivative of ∣r∣p|r|^p is p∣r∣p−1sgn⁡(r)p|r|^{p-1}\operatorname{sgn}(r). For 1<p<21<p<2, its magnitude grows sublinearly with ∣r∣|r|, compared with linear growth under squared loss. With residual normalization and loss weights held fixed, this reduces the relative contribution of extreme residuals to the loss gradient, providing a mechanism for mitigating their influence during training.

From a classical regression perspective, minimizing the population ℓpp\ell_p^p risk over unrestricted predictors yields a conditional pp-center under the required moment conditions. This recovers coordinate-wise conditional medians at p=1p=1 and the conditional mean at p=2p=2. Thus, pp adjusts the estimator’s sensitivity to large residuals beyond the fixed quadratic criterion used in Gaussian DDPM. Remark that Section 3.3 connects this perspective to our framework through matched divergence projection.

We additionally compare the training and sampling budgets of our model and DDPM at comparable PSNR. On CIFAR-10 with 75% contaminated training targets, our model achieves 36.600 dB with 30k training updates and 19 NFE, compared with DDPM’s 36.777 dB with 200k updates and 1000 NFE. We also include t-EDM trained for 100k updates and evaluated with 35 NFE.

ModelPredictionSamplerTraining updatesNFEPSNR ↑FID ↓Precision ↑Recall ↑
DDPMϵ\epsilonAncestral200k100036.777 ± 0.205268.460 ± 5.9840.103 ± 0.0070.480 ± 0.004
t-EDMEDMHeun100k3534.809 ± 0.266288.182 ± 0.5190.086 ± 0.0020.433 ± 0.039
p=1.2p=1.2, Dns.x0x_0Heun30k1936.600 ± 0.24785.828 ± 4.7480.374 ± 0.0100.463 ± 0.003

Regarding ν\nu, the supporting evidence discussed above is empirical, and we acknowledge this limitation in addressing the request for theoretical evidence. From a divergence-projection perspective, we hypothesize that an appropriately chosen finite ν\nu can reduce residual-distribution misspecification for heavy-tailed targets. Our current analysis, however, does not establish when this yields better distributional approximation or greater computational efficiency. Providing rigorous guarantees for these benefits remains a direction for future work.

[1] Kim et al. ∗t3*t^3-Variational Autoencoder: Learning Heavy-tailed Data with Student’s t and Power Divergence.* ICLR 2024.

[2] Pandey et al. Heavy-Tailed Diffusion Models. ICLR 2025.

Reviewer_odGPOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

This paper proposes a novel diffusion generative framework based on a family of the heavy-tailed distributions. The proposal derives two objectives based on the γ\gamma-power divergence naturally compatible with the heavy-tailed noise: an adjacent-state objective and a denoising objective , accompanied by different characterizations of the same path distribution for guiding generative path choice and providing sampling protocols. Theoretical analysis and experiments demonstrate the trade-off between preserving rare components and denoising robustness.

Critical strengths

The paper is well motivated by the diversity and imbalanced applications, and derives a principled generative construction that allows tractable learning and sampling from heavy-tailed distributions. The paper is guided by a unified and consistent design principle and lens through the joint-projection interpretation of the diffusion, and demonstrates the design considerations when we consider a different family of noise. Experiments align with the earlier motivations, are sufficiently diverse and well analyzed.

Critical weaknesses

My biggest concern is regarding the choice of presentation, which is the only reason I recommend a weak rejection at the current stage. I find that the paper is relatively dense in the theoretical constructions, and it requires the reader's significant efforts in extracting the core takeaways and the practical procedure to use the proposed generative framework. I recommend a substantial rewriting to prioritize and facilitate the intuitive understanding, and make the paper more accessible, fluent, and useful for the majority audience of the conference.

Questions for authors

The paper carefully characterizes the population-level differences between adjacent-state and denoising objectives. Meanwhile, recent work such as JiT suggests that even theoretically equivalent x0,ϵ, and vx_0, \epsilon, \text{ and } v-prediction parameterizations can behave quite differently in practice. Have the authors explored how these parameterization choices interact with the ℓp\ell_p loss and Adj./Dns. objectives? How does the practical numerical realization interact with the statistical differences?

6 comments

Thank you for recognizing the motivation, principled construction, and theoretical and empirical contributions of our work.

Response to Critical Weaknesses

We agree that the core ideas and practical procedures needed clearer explanation and have revised the manuscript accordingly:

  1. Clearer motivation and overview.

The Introduction now clarifies the motivation for heavy-tailed modeling and nonquadratic losses and states our central questions. In particular, the newly added Table 1 (also included in our global response) compares Gaussian DDPM with our ℓp\ell_p-power framework, while Eq. (1) introduces both objectives before their derivation.

  1. More detailed background explanations.

Section 2 now provides more detail on the joint-divergence interpretation of Gaussian diffusion (2.1), the compatibility of γ\gamma-power divergence with power-form heavy-tailed densities (2.2), and the Gaussian scale-mixture representation of the ℓp\ell_p-power family (2.3).

  1. Clearer theoretical takeaways.

Section 3 streamlines notation and results, presents the two objectives side by side, and explains why replacing a random target with its conditional mean can change the optimum under a non-quadratic loss.

Please also refer to our global response, which summarizes our main contributions and includes Table 1 comparing our framework with Gaussian DDPM.

Response to Questions

We conducted three additional experiments comparing x0x_0-, ϵ\epsilon-, and vv-prediction under the ℓp\ell_p loss and the adjacent-state (Adj.) and denoising (Dns.) objectives. The main findings are:

  1. Spiral generation: x0x_0-prediction better preserves the spiral as the embedding dimension increases. With this parameterization, Dns. produces more concentrated samples than Adj.

  2. Conditional spiral restoration: ℓ1.2\ell_{1.2}-Dns. shows smaller PSNR variation across parameterizations than Gaussian diffusion for D≤32D\leq32, but this advantage does not persist at D=64D=64.

  3. CIFAR-10: ℓ1.2\ell_{1.2}-Dns. achieves lower FID than both reported DDPM sampler configurations across all three predictors, while the PSNR comparison depends on the parameterization and sampler.

Before presenting the experimental details, we summarize the parameterizations below, with ct:=μt2+σt2c_t:=\mu_t^2+\sigma_t^2 and Δt:=Xt−1μt−1−X0\Delta_t:=\frac{X_{t-1}}{\mu_{t-1}}-X_0.

x0x_0-predϵ\epsilon-predvv-pred
Network predictionx^0,θ:=net⁡θ(Xt,t)\widehat x_{0,\theta}:=\operatorname{net}_\theta(X_t,t)ϵ^θ:=net⁡θ(Xt,t)\widehat\epsilon_\theta:=\operatorname{net}_\theta(X_t,t)v^θ:=net⁡θ(Xt,t)\widehat v_\theta:=\operatorname{net}_\theta(X_t,t)
Adj. targetX0+ΔtX_0+\Delta_tϵt−μtσtΔt\epsilon_t-\frac{\mu_t}{\sigma_t}\Delta_tvt−ctσtΔtv_t-\frac{c_t}{\sigma_t}\Delta_t
Dns. targetX0X_0ϵt\epsilon_tvtv_t
Equivalent loss weightwt,pw_{t,p}wt,p(σtμt)pw_{t,p}\left(\frac{\sigma_t}{\mu_t}\right)^pwt,p(σtct)pw_{t,p}\left(\frac{\sigma_t}{c_t}\right)^p

1. Generating a low-dimensional spiral embedded in a high-dimensional space.

Motivated by JiT [1, Fig. 2], we embed a 2D spiral in D∈2,8,16,512D\in\\{2,8,16,512\\} dimensions and compare all three parameterizations under both objectives.

For both objectives, x0x_0-prediction better preserves the spiral as DD increases, whereas ϵ\epsilon- and vv-prediction deteriorate across the tested pp values. With x0x_0-prediction, Dns. samples concentrate more closely around the spiral, while Adj. samples are more dispersed, particularly near its outer portion. Adj. dispersion varies with pp, whereas Dns. remains qualitatively similar across tested values.

We will include the corresponding spiral-generation figures in the appendix of the revised manuscript.

[1] Tianhong Li and Kaiming He. Back to Basics: Let Denoising Generative Models Denoise. CVPR 2026.

Response to Questions (cont’d)

2. Conditional restoration of the embedded spiral.

We compare ℓ1.2\ell_{1.2} and Gaussian diffusion under Dns. The network receives (Xt,y,t)(X_t,y,t), where yy is formed by adding Gaussian noise (std. 0.35 per coordinate) to the 2D spiral coordinates with probability 0.75 before embedding. We report PSNR over three seeds; “Obs.” evaluates the unmodified yy against the clean target X0X_0.

DDObs.ℓ1.2,x0\ell_{1.2},x_0ℓ1.2,ϵ\ell_{1.2},\epsilonℓ1.2,v\ell_{1.2},vGaussian, x0x_0Gaussian, ϵ\epsilonGaussian, vv
27.197.46 ± 0.097.30 ± 0.087.35 ± 0.067.42 ± 0.046.88 ± 0.106.82 ± 0.14
167.237.70 ± 0.047.02 ± 0.196.98 ± 0.157.44 ± 0.126.62 ± 0.176.77 ± 0.16
327.147.08 ± 0.156.72 ± 0.116.73 ± 0.096.87 ± 0.086.32 ± 0.215.91 ± 0.10
647.137.21 ± 0.174.96 ± 0.095.98 ± 0.047.10 ± 0.155.80 ± 0.025.66 ± 0.01

For D≤32D\leq32, ℓ1.2\ell_{1.2}-Dns. achieves higher mean PSNR for each parameterization and smaller variation across parameterizations than Gaussian diffusion. At D=64D=64, however, its variation exceeds that of Gaussian diffusion. For both methods, x0x_0-prediction yields the highest mean PSNR at all reported dimensions.

3. CIFAR-10 restoration under contamination.

We further compare the three parameterizations on CIFAR-10 with contamination probability ρ=0.75\rho=0.75.

PredictorMetricℓ1.2\ell_{1.2} Dns.ℓ1.2\ell_{1.2} Adj.DDPM (Heun, 39 NFE)DDPM (ancestral, 20 NFE)
x0x_0PSNR ↑36.600 ± 0.24735.204 ± 0.44132.370 ± 4.89634.768 ± 2.053
FID ↓85.828 ± 4.748302.162 ± 22.469351.657 ± 31.491334.522 ± 42.796
ϵ\epsilonPSNR ↑34.796 ± 0.18710.038 ± 4.44331.189 ± 1.09337.119 ± 0.030
FID ↓69.511 ± 6.444270.961 ± 100.913137.574 ± 32.718190.550 ± 8.667
vvPSNR ↑35.979 ± 0.11336.316 ± 0.10636.436 ± 0.22736.757 ± 0.228
FID ↓77.037 ± 2.812276.257 ± 8.523256.102 ± 20.202325.041 ± 33.267

ℓ1.2\ell_{1.2}-Dns. achieves lower mean FID than both DDPM configurations for every predictor and exhibits less variation across predictors than Adj. Its PSNR advantage, however, depends on the parameterization and sampler.

We use x0x_0-prediction for ℓ1.2\ell_{1.2}-Dns. based on its highest PSNR and our qualitative assessment, although ϵ\epsilon-prediction achieves the lowest FID. For DDPM, we use ϵ\epsilon-prediction because it provides the lowest FID with both samplers and the highest PSNR with ancestral sampling.

We hope that our response adequately addresses your questions and that our revisions make the paper clearer and easier to follow.

I appreciate the authors' efforts in revising a more accessible presentation and for carrying out additional experiments testing the interactions between the statistically different objectives and numerical realization of prediction heads.

For the specific new experiments, I am more interested in how the authors' own words would reason about and interpret the observed evidence in more depth. For example, it seems surprising that ϵ\epsilon-pred improves DDPM FID significantly while resulting in much higher variance for the heavy-tail objectives. As I skim your responses to other reviewers, I also find that the combinations of different numerical design factors can lead to dramatically different metric values even under a same objective. I understand that it can be demanding to finetune and refine the realization for every specific design, but these new results do increase my concerns regarding how to interpret the meaningful statistical distinctions of the proposed two objectives and the genuine benefits of the heavy-tailed proposal from the downstream quantitative performance.

While I would want to merit the authors' substantial refinement, I am currently maintaining my score for the above reasons but will keep the potential of increasing.

Comments on the paper

1 comment

Global Responses

Thank you to all reviewers for the thoughtful feedback. During the rebuttal period, we substantially revised the manuscript and conducted additional experiments to address concerns about presentation, empirical scope, and the interpretation of our results.

1. Clearer presentation and theoretical motivation.

Following Reviewer odGP’s suggestion, we revised the Introduction to clarify the logical progression of our framework and explicitly summarize its main contributions:

  • GSM characterization: We establish that the ℓp\ell_p-power family admits a GSM representation if and only if 0<p≤20<p\leq2.
  • Diffusion path construction: We construct GSM-based heavy-tailed paths with tractable distributions for training and sampling.
  • Variational objectives: We derive Adj. through joint γ\gamma-power minimization and characterize its equivalence to Dns. at p=2p=2 and possible differences in population optima for 1≤p<21\leq p<2.
  • Statistical analysis and empirical evaluation: We analyze rare-component sensitivity and conditions for improved denoising robustness with intermediate pp, alongside experiments on rare-mode preservation and denoising.

To clarify the relationship between our framework and Gaussian DDPM, we summarize their key features in Table 1:

AspectGaussian DDPMℓp\ell_p-power diffusion (this work)
Noise familyGaussian (ν=∞, p=2\nu=\infty,\ p=2)ℓp\ell_p-power (p<ν<∞, 1≤p≤2)(p<\nu<\infty,\ 1\leq p\leq 2)
Forward pathXt=dμtX0+σtεtX_t\overset{\mathrm d}{=}\mu_tX_0+\sigma_t\varepsilon_tXt=dμtX0+σtΛt−1/2εtX_t\overset{\mathrm d}{=}\mu_tX_0+\sigma_t\Lambda_t^{-1/2}\varepsilon_t
DivergenceKL divergenceγ\gamma-power divergence
Adj.–Dns. relationEquivalentEquivalent for p=2p=2; not generally equivalent for p<2p<2.

Given the time constraints during the rebuttal period, we prioritized revisions to the main manuscript. Some appendix sections may therefore still reflect the earlier version, and not all additional experiments reported in our responses have been incorporated. We will update the appendix to ensure consistency with the revised main text and include the remaining experimental results.

2. Additional experiments and ablations.

In response to Reviewer Un68, we expanded the evaluation to clean generation, varying contamination levels, and class-imbalanced datasets, and added larger-budget baseline comparisons. We also conducted prediction-parameterization ablations in response to Reviewer odGP and sampler and noise–loss ablations in response to Reviewer S3Hd.

Detailed results are provided in the individual responses. We hope that our response adequately addresses the reviewers’ questions and that the revisions make the paper clearer and easier to follow.

update additional experiments on the manuscript

We have incorporated additional experiments conducted during the rebuttal period into the main manuscript and appendices. In particular, we have replaced the synthetic rare-mode coverage experiments in the main text with the CIFAR-10-LT and CIFAR-100-LT experiments. Due to the limited time available, some experimental details and implementation specifications remain to be added. If OpenReview permits further revisions after the discussion period, we will complete these updates.

We thank all reviewers for their constructive feedback and continued engagement, which have helped us substantially improve the manuscript since the initial submission.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.