acceptodds
Under review as a conference paper at ICLR 2027

Fine-Tuning Generative Models for Extreme Events via CVaR-Penalized Wasserstein Gradient Flows

Abstract

In many high-stakes domains, extreme events carry substantial consequences, yet learning the heavy-tailed distributions that govern them from finite samples remains challenging: the quantities of interest are driven by a few extreme observations, so the tail region is under-sampled relative to its importance. Even generative models tailored for heavy tails capture the tail region inadequately in practice. We propose the Conditional Value-at-Risk (CVaR)-penalized Generative Particle Algorithm (CVaR-GPA), a tail-agnostic algorithm for fine-tuning generative models toward heavy-tailed targets, built as a time discretization of the Wasserstein gradient flow of the Lipschitz-regularized KL divergence penalized by a CVaR discrepancy term. Such a flow can be initialized from the output samples of the pre-trained model, which are then transported along the gradient descent of the loss functional, without requiring access to the pre-trained model's internal architecture. The Lipschitz-regularized KL divergence requires minimal assumptions on the target, while the CVaR penalty focuses the flow on the tail. The CVaR penalty depends on the target only through a scalar tail statistic, inducing a velocity field that remains active in the under-sampled tail region at a dimension-free estimation cost. The resulting velocity field, and hence the depth of the transport map, implicitly adapts to the target distribution, without any target-specific modifications. Across four targets, including two real-world, high-dimensional datasets (daily streamflow in the Ohio River basin () and the Fama–French portfolios () with tail indices ranging from to ), fine-tuning with CVaR-GPA reduces global and tail errors by geometric-mean factors of and , respectively, across seven pre-trained models spanning GANs, diffusion models, and other generative flows, with a single set of hyperparameters.

Primary area:
generative models
Keywords:
Extreme events, Wasserstein gradient flows, heavy-tailed distributions, Conditional Value-at-Risk, Lipschitz-regularized divergences, particle neural algorithms, fine-tuning

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_ftR7OpenReview ↗
Rating
3
Confidence
3
Informativeness
3(+1)
Presentation
2
Soundness
3
Read review
Paper summary

The paper proposes to learn a generative model for extreme events via post-training of pre-trained models rather than via a new pre-training technique. This process is formalized via a form of distributional adaptation formalized via a Wasserstein gradient flow and executed by exploiting the variational formulation of the CVaR. The authors provide a Theorem that characterizes the gradient flow's velocity field, and they provide an experimental assessment of the proposed algorithm on both toy and real-world data, showing promising results in both settings.

Critical strengths
  1. The idea of learning a better model for extreme events via fine-tuning rather than pre-training seems original and interesting. I believe this is a promising direction that might influence subsequent methods.
  2. The proposed algorithm is principled and seems to improve models in practice on toy and small-scale domains. I believe this performance is promising for the proposed methodology and that future works could scale this idea further to more complex and higher-dim. setting.
Critical weaknesses
  1. (Algorithm Question) Recent works for flow and diffusion models are able to approximately execute flows in probability space without using empirical estimates of the density. But since this method seem to work also for other generative models, where a velocity field is not necessarily present (eg GANs) I am confused about its workings and have the following key question: Is the empirical distribution \hat{Q}_k ever directly estimated? Related to this, I found the presentation of the algorithm rather unclear from a procedural, computational standpoint. I suggest to replace the Equations in page 6 with a clear pseudocode where every step is explicitly written, including sampling and estimation of quantities.
  2. (Experiments) Experiments seem quite toysh/low-dim. While they can be of support for a theory paper, they certainly do not show convincing results for most applications in generative modeling, where typical design spaces are higher-dimensional than the ones showcased here by multiple orders of magnitude.
  3. (Framing agains prior literature) The proposed method seems to be a case of distributional fine-tuning as introduced in [1]. In particular, besides the different geometry of the gradient flow, the formal problem presented of CVaR-regularized divergence minimization appears to be a sub-case of the general problem introduced there for arbitrary divergences and functionals on probability measures induced by a generative model. Moreover, the specific variational expression of CVaR employed in this work has been used by prior work to adapt distributionally generative models to control CVaR functionals [2]. These prior works should likely be presented early on, rather then kept in the appendix, since the proposed methodology seems to directly build on them.

References

[1] Flow density control: Generative optimization beyond entropy-regularized fine-tuning. De Santi et al., 2025 NeurIPS

[2] Efficient tail-aware generative optimization via flow model fine-tuning. Wang et al., 2026 ICML

Questions for authors
  1. See the question in Weakness 1
  2. Is this the first method that studies how to obtain a more heavy-tailed model (or any other formalization of extreme event modeling) via post-training / fine-tuning of a pre-trained generative model?
Reviewer_yzjfOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

This paper presents a means of editing the samples from generative models to be more inclusive of the tails in heavy-tailed distributions. If I understand properly (I'd appreciate confirmation from the authors during the rebuttal period), the basic idea is to take a collection of points that have been output by a generative model, and then to flow those points according to a regularized Wasserstein/CVaR/KL gradient flow to improve the collection.

The theory in the paper is fairly comprehensive (if hard to read) but the experiments are on the superficial side. If I understand properly, the model was only evaluated on two datasets of medium dimension (25, 64), as well as some synthetic distributions. So the value here is mostly in the construction itself.

Critical strengths

It seems like the majority of methods intended to address sampling from heavy-tailed distributions do so by fine-tuning the model itself. This method (again, if I understand it properly) stands out for acting on the collection of samples a posteriori. This is an interesting idea with clear trade-offs --- on the one hand we don't have to retrain the basic model, on the other we need to generate an entire collection of samples and run a gradient flow to draw samples end-to-end.

The paper aligns with modern trends in applying optimal transport and Wasserstein gradient flows and seems to be relatively simple (although I'd appreciate a few timing numbers to give an idea of how long the algorithm takes to run). The experiments that are present here seem to indicate that the algorithm works as predicted.

Critical weaknesses

My two main complaints involve the exposition of the paper and level of experimental validation.

On the exposition side, the paper needs a substantial overhaul. The text is extremely dense mathematically and written in a way that requires intimate knowledge of recent work by a certain subcommunity in ML. An example that stands out to me is the paragraphs surrounding eq(1), which are laden with undefined mathematical terms. "CVaR" itself appears in the title of the paper and the introduction, but isn't defined until (4) and is presented without much intuition. I found myself searching online for definitions of CVaR, which seem to vary from source to source and are often informal or 1D (targeted to business rather than stats/ML/math audiences).

The notation in the paper is also quite complex and doesn't aid understanding. For example, the parameter alpha seems to be critical but I couldn't really figure out its role. The "risk function" g appears many times, but I wasn't sure how to map g onto a standard ML pipeline (is it a loss function?). Equations (4-5) and the unnumbered equation on line 213 are messy, with two superscripts and two subscripts, requiring the reader to remember quite a bit; and moreover, the final of the three apparently applies to empirical distributions but I see no samples or sums (g is a function, right? so, what goes into g?). I might suggest providing a few simple examples to contextualize the development, which is quite abstract.

Some of the math/modeling decisions are presented as "fait accompli" without justification/intuition (cf "equally essential" on line 232, for example). The text needs to be carefully revised paragraph-by-paragraph to make sure every single equation, loss function, model, and so on is well-justified, rather than just presenting the method as a sequence of steps.

On the experimental side, I'd like to see much more comprehensive validation. It seems the paper only includes two real-world experiments. These experiments are tested on many values of parameters and settings, which is a positive, but a wider variety of applications---and higher dimensionality---would help a lot. For example, note the standard in image generation (not necessarily the application needed in this paper, but a point of reference) has thousands of dimensions and much larger datasets (CIFAR, ImageNet, etc.). I understand the goal here isn't to get perfect samples but it would be good to make sure the method is flexible and that the real world examples aren't cherry picked.

Smaller comments: The writing of the "AI Use Statement" is itself quite messy (which perhaps indicates the authors didn't use AI ;-) ). The first sentence has an oddly placed "not" that was quite awkward.

Questions for authors

Is my characterization of your method in the first box correct?

One specific math concern: My understanding is that barycenter problems are trivial both in 1-Wasserstein and in KL divergence, basically given by pointwise formulas (for example, W1 interpolation is well known to yield "mass teleportation artifacts," in contrast to the geometric displacement interpolant obtained by Wp for p>1). Eq (2) seems to be a barycenter between W1 and KL, so I'd be concerned that the optimal gamma is just some simple formula. Is this the case? And if not, what's the intuition why this combination of W1 and KL leads to more interesting/geometric behavior than either one by itself?

Reviewer_ohZ3OpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

The paper introduces CVaR-GPA, which is an approach that transports samples from a pre-trained model to a heavy-tailed target. The core idea is to discretize the Wasserstein gradient flow with an objective that balances KL loss and CVaR. The authors provide theoretical results/bounds, and empirically evaluate CVaR-GPA.

Critical strengths
  • Theoretical results: The theoretical results (variational analysis, subgradient analysis, etc) are relevant and substantial. Although I did check the math/derivations line by line, I find the overall big picture conveyed to be sensible. The presentation, though, can be improved to make it easier for the readers.
  • Methodological innovation: I find the idea of combining KL with a CVaR penalty to be simple and clever, and the method tackles an important problem in generative modeling with heavy tails, which is an area of interest to the ICLR community, and can have real implications for applications in finance.
Critical weaknesses
  • Clarity/Presentation: The writing of the current article is rather dense, and it requires substantial background in different areas of mathematics and generative modeling for the reader to understand. I understand that this is partly unavoidable given the nature of the method which involves wasserstein gradient flows and subgradients, but I suggest delegating a lot of the notational setup and details from the main text to the supplement, and focusing more on the big picture and take aways in the main text.
Questions for authors

There are some highly relevant missing citations that pertain to tails in generative modeling. I list a few of them below:

  • Copula & marginal flows: Disentangling the marginal from its joint. Wiese et al.
  • On the Statistical Capacity of Deep Generative Models. Tam and Dunson
  • On some theoretical limitations of generative adversarial networks. Oriol and Miot.

Comments on the paper

1 comment

Dear reviewers,

The author-reviewer discussion period is ending soon. If you have not done so yet, please read the authors’ rebuttal and reply to it. Tell the authors whether it addresses your concerns, and update your score if your assessment has changed. With the high number of submissions this year, your input matters more than ever.

Thank you for your time and help.

Best regards,


Your Area Chair

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.