- Rating
- 2
- Confidence
- 2
- Informativeness
- 3
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper develops a variational framework for heavy-tailed generative modeling using an -power family that includes Student-t and symmetric Pareto distributions. It characterizes Gaussian scale-mixture representations and constructs temporally coupled diffusion paths. For fixed decoder scales, joint -power-divergence projection yields an adjacent-state -power prediction loss. The paper shows that replacing the adjacent target with the clean-data target preserves the objective up to an additive constant at , but generally changes the objective and its population minimizer below .
The analysis relates these minimizers to conditional Fréchet -centers and identifies conditions governing rare-component sensitivity and reconstruction risk. Experiments include imbalanced heavy-tailed mixtures, character-level typo correction on WikiText-103, a controlled colored-MNIST estimator study, and image restoration and generation from contaminated training sets. The results support a distinction between preserving valid rare structure and suppressing corrupted targets. The exact variational construction concerns a discrete stochastic model; the deterministic sampler used in the main experiments is explicitly acknowledged as heuristic.
Critical strengths
-
A consequential distinction between training targets. Theorem 3.4 derives an adjacent-state objective from a joint path projection without assuming a KL-style additive chain rule. Theorem 3.6 then establishes the special role of the quadratic case and the general failure of objective equivalence below it. Proposition 3.7 provides a mechanism for different responses to rare conditional components, while Corollary 3.8 identifies a condition under which greater sensitivity increases reconstruction risk. These results make the contribution more substantial than introducing an intermediate loss exponent. The family and scale-mixture ingredients build on established work; the valuable contribution is their integration into a path objective and the analysis of the prediction targets.
-
Substantial controlled validation with explicit boundaries. Appendices F.2 and F.3 use matched architectures and training conditions and report results over three seeds. The noise–loss ablations in Tables 6–7 and the dimension/noise sweep in Table 3 directly probe the proposed explanation. At , adjacent training reduces the mixture discrepancy from 0.062 to 0.012 relative to matched denoising training (Table 2). The image results show substantial improvements under the specified contaminated-training regime; for example, CelebA FID is 37.176 for denoising versus 267.229 for DDPM (Table 5). Appendix G also acknowledges conflicting fidelity–coverage rankings and the limits of the theoretical explanation. This combination constitutes a meaningful body of work.
Critical weaknesses
-
The exact model and the empirically evaluated sampler are not sufficiently connected. Theorem 3.4 concerns the coupled discrete reverse model, whose exact stochastic sampler maintains an accumulated residual statistic. However, the mixture and image experiments use the deterministic ODE branch. Appendices E.3 and G correctly acknowledge that a conditional -center generally does not identify a conditional-mean probability-flow velocity, so marginal preservation does not follow from the discrete projection. The acknowledgement is valuable, but the paper does not empirically establish how results change when using the exact stochastic model, the ODE heuristic, or direct prediction. This limits the extent to which the application results validate the variational construction, while leaving the discrete theoretical contribution intact.
-
Application-level gains are less well isolated than the synthetic objective comparison. Table 7 already shows that Student-t noise combined with the same nonquadratic penalty can perform similarly to the matched -power family: at , their adjacent-training discrepancies are 0.013 and 0.012. Yet the image experiments do not include the corresponding Gaussian/Student-t controls with the same penalty, prediction parameterization, and sampling procedure. Appendix F.5 also changes prediction parameterization and training weighting across methods. Consequently, the large image gains cannot yet be cleanly attributed to the new noise family rather than robust loss design and other choices. Robust diffusion objectives are acknowledged in Appendix A but are not directly evaluated in this setting.
The language experiment presents another attribution issue: at , head-word recall falls from 0.991 under adjacent training to 0.103 under denoising training (Table 4). This broad degradation exceeds a selective loss of rare words. Appendix G acknowledges possible effects of learned embeddings, auxiliary classification, and optimization, but leaves them unresolved. These limitations reduce the confidence with which the application differences can be attributed to the population-level mechanism.
Questions for authors
-
Can you compare the exact stochastic branch of Algorithm 1, the deterministic ODE branch, and direct prediction using the same trained predictors on a representative setting? Does the adjacent-versus-denoising ordering survive the sampler choice? This would materially affect the connection between the variational construction and the reported practical behavior.
-
Do the restoration and generation gains remain when Gaussian or Student-t perturbations use the same nonquadratic loss, prediction parameterization, and comparable sampler? The synthetic controls in Table 7 are informative; the corresponding application-level control would clarify the added practical value of the -power family. A comparison with an appropriate robust diffusion loss would also help establish this distinction.
-
What explains the severe head-word degradation of the denoising model in Table 4? Can the existing training diagnostics or a control with fixed embeddings and matched auxiliary-loss weighting distinguish a target-induced statistical effect from an optimization or representation-learning failure? The answer could change how strongly I regard this experiment as evidence for the proposed mechanism.
-
Please specify the generated and reference image counts and feature-extraction protocol used for FID and precision/recall, and explain how the selected configurations in Table 9 were chosen. Appendix F.5 describes training in detail but does not make these evaluation choices equally explicit. This is a reproducibility clarification, not a request for additional application domains.
4 comments
Response to Question 3
We find that the previously reported head-word recall collapse of Dns. is resolved in the revised experiments using 32-bit floating-point precision and gradient clipping with a threshold of 20, although a substantial performance gap relative to Adj. remains on other metrics.
These changes were introduced to improve numerical stability. Gradient clipping is intended to control the large gradients that can arise under heavy-tailed perturbations. For Dns., head-word recall increases from 0.103 to 0.993, while increases from 0.068 to 0.604. These results indicate that the original collapse was sensitive to the numerical and optimization settings and should not be interpreted as direct evidence of the population-level mechanism.
Table C3. Updated language results using 32-bit floating-point precision and gradient clipping with a threshold of 20.
| Model | ↑ | Accuracy ↑ | Sentence accuracy ↑ | Head recall ↑ | Tail recall ↑ |
|---|---|---|---|---|---|
| Norvig-10 | 0.665 | 0.614 | 0.117 | 0.483 | 0.113 |
| , Adj. | 0.868 ± 0.008 | 0.891 ± 0.006 | 0.398 ± 0.009 | 0.998 ± 0.001 | 0.445 ± 0.009 |
| , Dns. | 0.604 ± 0.011 | 0.700 ± 0.006 | 0.050 ± 0.004 | 0.993 ± 0.002 | 0.112 ± 0.006 |
| , Adj. | 0.815 ± 0.008 | 0.857 ± 0.008 | 0.313 ± 0.008 | 0.989 ± 0.001 | 0.363 ± 0.008 |
| , Dns. | 0.500 ± 0.010 | 0.684 ± 0.009 | 0.033 ± 0.003 | 0.939 ± 0.004 | 0.085 ± 0.006 |
| t-EDM | 0.721 ± 0.004 | 0.735 ± 0.002 | 0.185 ± 0.005 | 0.988 ± 0.002 | 0.182 ± 0.003 |
| DDPM | 0.722 ± 0.003 | 0.754 ± 0.006 | 0.255 ± 0.003 | 0.971 ± 0.006 | 0.250 ± 0.005 |
At , both methods achieve high head-word recall (0.993 for Dns. and 0.998 for Adj.), but Adj. retains substantially higher tail-word recall (0.445 versus 0.112), , overall accuracy, and sentence accuracy. The Adj.–Dns. ordering also persists at .
The updated results support an empirical difference between the training targets under the revised numerical settings. However, they do not fully isolate the population-level mechanism from optimization and representation-learning effects. We will update Table 4 in the revised manuscript and the experimental description to report the use of 32-bit precision and gradient clipping.
Response to Question 4
We use a common evaluation protocol across datasets:
- FID: We compare 50,000 generated images with 50,000 clean reference images using Inception-V3 pool3 features with TensorFlow FID weights.
- Precision and recall: We use the neighborhood-based estimator of Kynkäänniemi et al. (2019) with , using the same Inception features and the same sets of 50,000 generated and 50,000 reference images.
- PSNR: We evaluate the test set of each dataset, including all 10,000 test images for CIFAR-10. For each test image, we use one corrupted observation, average restoration outputs, and clip the average to the valid image range before computing PSNR against the clean target.
The configurations in Table 9 were selected based on the PSNR–FID tradeoff, jointly considering reconstruction accuracy and distributional quality. We will explicitly document this selection criterion and the evaluation protocol in Appendix F.5.
We hope that these additional experiments and clarifications address the reviewer’s questions.
Thank you for the additional experiments and clarifications. The sampler comparisons, matched noise–loss controls, and numerical-stability checks address several of my original concerns. I also appreciate the clarification of the evaluation protocol.
The theoretical distinction between adjacent-state and denoising objectives remains interesting. However, the new controls do not establish a clear practical advantage for the proposed noise family, and the strongest application results still rely on the heuristic sampler. The connection between the variational construction and the demonstrated practical gains therefore remains limited.
Overall, I appreciate the authors’ efforts during the discussion and maintain my original rating of weak rejection.
Thank you for the follow-up. We would like to clarify the reasons for choosing the -power family and the role of sampler selection in our practical results.
1. Theoretical role of the proposed noise family
The proposed -power family provides a consistent connection between heavy-tailed paths, -power divergence, and the adjacent-state objective. Although the theoretically matched noise–loss combination need not achieve the best empirical performance in every setting, it provides a principled basis for deriving Adj. and analyzing how replacing its target with a conditional mean changes the optimal predictor under non-quadratic losses.
2. Exact stochastic sampling in the language experiments
Regarding sampling, we acknowledge that practical performance depends on the task and sampling procedure. Importantly, the language experiments use the exact stochastic Joint sampler, a detail omitted from the original experimental description. Under the updated numerical settings in Table C3, Adj. with achieves an score of 0.868 and tail-word recall of 0.445, compared with 0.722 and 0.250 for DDPM Thus, meaningful application gains are also obtained using the discrete reverse model specified in our construction.
3. Task-dependent sampler selection
For image generation, we acknowledge that our strongest results rely on the heuristic Heun sampler. However, this choice concerns inference and does not alter our training framework. A key practical consideration is whether stochastic perturbations during sampling are beneficial for the task. Just as Dns. is adopted as a surrogate training objective, sampler selection can also be viewed as a task-specific decision at inference. Taken together, these findings can be interpreted in terms of task-dependent sampler selection and the practical role of randomness.
Thank you for the careful and technically insightful review. Below, we provide additional experiments, explain their implications and limitations, and clarify the evaluation protocol.
Response to Weakness 1 & Question 1
We evaluated the sampling procedures on CIFAR-10 with 75% contaminated training targets. For each ℓ1.2 model and seed, all samplers use the same EMA checkpoint obtained after 30k training updates, with x0-prediction and no retraining between samplers. “Joint” denotes the exact stochastic branch of Algorithm 1 for the specified discrete reverse model, which maintains the accumulated residual statistic. We compare this branch with the deterministic Heun branch and direct prediction using a single network evaluation, and additionally report ancestral sampling.
For context, we also include DDPM and t-EDM baselines. Their regular runs use 50k training updates; the extended-budget runs, marked with *, use 200k updates for DDPM and 100k for t-EDM. NFE is reported per sampling trajectory.
Table C1. Sampling comparison on CIFAR-10 with 75% contaminated training targets.
Dns. achieves higher mean PSNR and lower mean FID than Adj. across all four samplers, although absolute performance varies substantially. Heun–Dns. achieves the lowest mean FID, while Joint and direct sampling show limited coverage. Ancestral Adj. has high recall but very low precision, highlighting the need to consider both metrics.
Response to Weakness 2 & Question 2
We added Gaussian and Student-t controls on CIFAR-10 with 75% contaminated training targets. All configurations in Table C2 use x0-prediction, the same loss weighting, 30k training updates, and Heun sampling with 19 NFE. We vary the noise family and residual penalty. Here, ℓ1.21.2 and ℓ22 denote the nonquadratic and squared residual penalties, respectively. The DDPM and t-EDM baseline configurations are reported separately in Table C1.
Table C2. Noise–loss controls with common prediction parameterization, loss weighting, training budget, and sampling procedure.
Replacing squared loss with the nonquadratic penalty improves all four reported mean metrics for both heavy-tailed noise families. With the penalty fixed, Student-t achieves the lowest mean FID and highest mean precision, while Gaussian noise achieves the highest mean PSNR and recall. The ℓ1.2-power model lies between them.
These controls highlight the contribution of nonquadratic losses and show that the gains cannot be attributed solely to the proposed noise family. Our framework connects the distributional construction to the training objective through matched divergence projection.