acceptodds
Withdrawn from ICLR 2027

JumpSampler: Making Distribution Tails Easier to Chase

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

14 views

Abstract

Real-world data distributions can exhibit heavy-tailed characteristics in both pixel and latent spaces, assigning non-negligible probability mass to rare or low-density regions. However, diffusion and flow-based generative models rely predominantly on local, continuous sampling dynamics, which can be insufficient for capturing such heavy-tailed structures, potentially leading to under-represented tail events, reduced sample diversity, and distorted distributional statistics. In this paper, we provide a theoretical characterization of this limitation. Specifically, we show that, under standard regularity conditions, most samplers produce light-tailed terminal distributions, whereas sparse heavy-tailed perturbations can induce polynomial tail behavior while keeping the deviation from the original sampler controlled. Therefore, we design **JumpSampler**, a training-free and plug-in sampling strategy that augments pretrained diffusion and flow models with jumps, enabling occasional nonlocal transitions while remaining broadly compatible with diverse sampling schemes and generative architectures. The key design is to inject heavy-tailed perturbations only at sparse sampling steps and progressively anneal their effect over the sampling trajectory, allowing the sampler to explore distant low-density regions while preserving local refinement near the end of generation. We theoretically show that such sparse heavy-tailed jumps induce polynomial tail behavior in the resulting distribution, and further establish stability guarantees showing that the jump-augmented sampler remains controlled in both total variation and Wasserstein distance from the original sampling process. Extensive experiments demonstrate that JumpSampler improves tail fitting, distributional matching, and overall generation quality across both stochastic and deterministic samplers.

Primary area:
generative models
Keywords:
generative model

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_HVRjOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
3
Soundness
1
Read review
Paper summary

This paper modifies generative diffusion or flow by adding during the generative process some heavy-tailed distributed jumps. The aim is to generate heavy-tailed data in a way that gives a better sampling of the tails. The paper proposes a procedure for adding these random jumps at random times in uniformly random directions; this procedure depends on the power nu defining the heavy tail and two parameters lambda, controlling the rate of jumps, and beta, controlling how this rate evolves with the generative time. The theoretical part proves that this procedure generates the heavy tails, with the exponent which is the one injected in the jumps. It also proves that the jumps do not distort too much the generated distribution. The experimental part first tests the procedure on some toy distribution in one dimension, and then on images. For each image dataset , picking up some appropriate values of the parameters nu, beta and lambda it provides some some evidence of a rather small improvement in IS and FID and ECS, with respect to the standard procedure.

Critical strengths

The procedure of jumps is well defined , it induces heavy tails, and it generates a distribution which does not differ too much from the distribution obtained without jumps

Critical weaknesses

The jumps are not correlated with the data structure. If we focus on images, the paper never measures on the data what is the quantity that ihas a power law distribution and that it aims at reproducing. The procedure uses isotropic jumps which cannot reproduce specific properties of data, unless data would have isotropic tails. The directions of jumps their frequencies and their amplitudes should be learned from data. Here the direction is uniformly random, and the parameters nu, lambda, beta are chosen a priori, or optimized over many trials.

The theoretical part is correct but it basically proves expected results: that a power law distributed jumps create power-law tails, and that the distances between the distributions generated with or without jumps is not too large. It does not address the crucial question: What is the property of data that one wants to reproduce? Does the new method with jumps provide an improvement? The one-dimensional toy example is too simple: with only one variable the isotropy of jumps is no longer a problem.

The experimental part on images finds improvements in FID, IS, ECS which are really small. There is no evidence of improvement on a crucial tail property.

Questions for authors

Could you give an example of a property that you expect has a heavy tail distribution, measure it empirically on the image dataset that you train your model on, and measure the same property for new images generated by the standard procedure on the one hand, and for new images generated by your jump procedure on the other hand.

How do you chose the parameters nu, lambda, beta for each dataset

Reviewer_oBB3OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

The paper proposes a modification to the sampling procedure of diffusion models by introducing random jumps. The authors justify it by observing that real-world distributions may be heavy-tailed, while, by construction, diffusion models are not able to approximate these heavy tails well. To address this, the authors suggest a simple scheme that helps to recover polynomial tails of a target distribution. They provide a theoretical analysis of the proposed scheme and test its performance on image generation tasks.

Critical strengths

I think the key strengths of the paper are:

  1. an interesting perspective on a non-obvious failure mode of a diffusion generative process. The authors show that using a standard reverse-time SDE (Equation 3) for generation may fail to recover the tails of the target distribution;
  2. the authors provide a theoretical analysis that shows that the distribution that emerges as a result of a generative process is necessarily sub-Gaussian, meaning that, if the target distribution had exponential or polynomial tails, they wouldn't be recovered;
  3. the scheme the authors propose to remedy the problem is training-free and easy to implement.

Moreover, the authors show that the improved algorithm achieves slightly better performance on some image generation benchmarks.

Critical weaknesses

Comments on the theoretical analysis

  1. Please make the theorem statements self-contained: Theorem 1 uses X00X_0^0 and rr but does not define them, and the same holds for Theorem 2. Theorem 3 does not define L\mathcal{L}, λ\lambda, β\beta and TV\mathrm{TV}; the same holds for Theorem 4. Although it is more or less clear what is meant in the theorems, I believe these suggestions could significantly improve the clarity of the presentation.
  2. Little discussion is offered after Theorems 3 and 4. How should I interpret the bounds in Equation 20 and in Corollary 1? I appreciated the text in lines 371–377, but I would suggest adding more discussion right after each theorem to help the reader better understand the importance of each result.

Comments on the experiments

My major concern about this paper is that I do not find the experiments convincing enough. Although I acknowledge the results in Tables 1, 2, and 3, it is quite difficult for me to believe that the improvements offered by the algorithm are substantial. The FID, IS and sFID values are not considerably different, and the absence of standard deviations makes it hard to distinguish an algorithmic improvement from a mere numerical difference. For example, the FID values in Table 1 are 1.80 vs 1.76 or 1.54 vs 1.53. More broadly, I wonder whether image generation is the best problem to showcase the benefits of a heavy-tail-aware diffusion sampling method. Although the authors offer some analysis of the issue in the form of Proposition 1, I am unconvinced that heavy tails are that much of an issue for latent diffusion. Perhaps some numerical experiments that reveal what exactly goes missing when heavy tails are ignored would help. Having said that, I acknowledge that showing the advantage of a heavy-tail-aware diffusion sampling method is a challenging task. I list some suggestions for additional experiments below.

Additional comments on the paper

  • I venture to suggest moving Algorithm 1 to page 5, as this is where the authors discuss it.
Questions for authors

Here is what I think could possible better showcase the advantage of the method proposed in this paper.

  1. For Tables 1, 2, and 3, compute the metrics using several seed (e.g. 5) and present the average together with standard deviation.
  2. I would suggest moving the experiment on a synthetic target from Appendix C to the main text. This experiment is a nice way to motivate your method. Besides I would suggest another synthetic experiment: use a mixture of Cauchy distribution in 2, 10, 50, 100, 500, 1000 dimensions, train a diffusion model on them compare deterministic sampler, stochastic sampler and JumpSampler on each of the targets. It could be interesting to see if how the performance varies as dimensionality grows.
  3. Lastly, I suggest testing the JumpSampler on distribution that are known to have heavy tails. I suggest looking into some drug discovery, protein generation, or molecular generation problem. Unfortunately, I struggle to name a specific problem right now.

I would suggest doing (1) and (2) during the rebuttal period and keep (3) for future work.

Moreover, if I am correct, the problem studied in the paper also affects more advanced sampling schemes, such as Heun, DPM++ and GLASS flows. I find this observation very important and would welcome a more detailed discussion of it in the paper, ideally supported by numerical experiments.

Reviewer_5YsGOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper addresses the problem that real-world data can be heavy-tailed, whereas standard diffusion and flow samplers are driven by local Gaussian dynamics and tend to produce light-tailed outputs. The authors propose JumpSampler - a training-free, plug-in modification that augments a pretrained sampler with sparse heavy-tailed (Student-t) jumps along the trajectory. The jumps are annealed, so they are more probable at the high-noise beginning and vanish towards the end. The paper provides a theoretical characterization (light-tailed base sampler vs. polynomial tail of JumpSampler) with TV and Wasserstein stability bounds, and evaluates on a 1D toy target and several image generation settings.

Critical strengths

The paper has a few significant strengths overall, which I will outline below.

Strengths:

  1. The core idea - combining the pretrained diffusion/flow dynamics with jump (Lévy-type) perturbations - is mathematically reasonable and well-grounded in the jump-diffusion literature.
  2. The method is training-free and plug-in, with no additional network evaluations, and is compatible with a wide range of samplers and architectures.
  3. The theoretical part is clean, and the stability results tie the distributional deviation to interpretable hyperparameters.
Critical weaknesses

However, despite its strengths, the paper has a few major and minor weaknesses.

Major weaknesses:

  1. My main concern is the empirical significance of the method. The large improvements are shown basically only on the 1D synthetic toy target, where the actual tail-specific metrics are reported. On the real data the gains are marginal and within what looks like a noise. Moreover, FID and ECS are global feature-space metrics and don’t measure the tail recovery directly, so the paper never demonstrates on the real images the very thing it motivates. I would like to see the tail-specific metrics computed on the real data.
  2. Related to the above, the ablation ranges are too narrow to be convincing - the checked hyperparameters are very close to each other and the resulting differences are tiny, so it is hard to tell whether the jumps have any real influence. Please, could you check more distant settings?
  3. The novelty is not fully clear to me. Jump-diffusion for sampling is known, and the heavy-tailed generative modeling is already covered in the related work; the main new angle seems to be the training-free framing. Besides, the related RL/entropy-based diffusion samplers (designed exactly for a better exploration across the space) are not discussed nor compared, whereas they look relevant to the motivation.
  4. I have a conceptual concern regarding the annealing. Since the jumps are concentrated at the high-noise beginning and vanish towards the end, and the base sampler then maps the perturbed state in its usual way, I’m not convinced that these early jumps really reach new tail regions and modes, rather than being re-absorbed by the subsequent local dynamics. An ablation isolating where in the trajectory the jumps actually help would be much more convincing.
  5. Theorem 2 only guarantees a polynomial tail with the exponent set by the hyperparameter ν, not matched to the target. So the method can impose a heavy tail, but there is no mechanism ensuring the imposed exponent agrees with the real target tail.

Minor weaknesses:

  1. The method relies on a few engineering choices somewhat hidden behind the clean jump-diffusion story (e.g., the clipping of the activation probability, the fixed schedules/defaults). Would be good to discuss the sensitivity to these.
  2. Some symbols in the regularity assumptions are introduced tersely / used before being clearly defined in the main body, which slightly hurts the readability.
Questions for authors

I would like to see especially the responses to the issues mentioned as weaknesses. Besides, I have a few additional questions:

  1. What about comparing (or at least discussing) the RL-based / entropy-based diffusion samplers that allow for a better searching across the space? Have you checked any of them?
  2. How heavy-tailed the real-world data actually is? It would be great to have some direct measurement of the tails on the real latents/pixels that motivate the method.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.