- Rating
- 3(+1)
- Confidence
- 4
- Informativeness
- 3
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
The paper deals with the problem of topological constraints in the normalizing flow framework. It has well known, that there is a Lipschitz tradeoff in handling multimodal (or more generally topologically different) distributions from e.g. a standard Gaussian. The authors deal with this issue with defining a graph cover, which essentially defines a graph structure on the data points and then finds an overlapping partition of such which is then used to learn a mixture model. Different ways (cover versus learning a mixture model) are ablated in the paper.
Critical strengths
-
I think that the framework of graph covers is genuinely novel and the right stab at the problem. It is very reminiscent of [Cornish et al], but a good new rephrasing of this problem.
-
The numerical results are encouraging, and are a clean ablation that help a deeper understanding isntead of improving just numbers.
Critical weaknesses
-
There is a reason why this line of literature has received less attention in the last years. Of course when learning a NF there is a topological mismatch, but the question is whether it hinders training/inference. the rise of diffusion and flow matching models split up the prolbem into smart subproblems, see e.g., [1]. This allows making deeper networks that still train nicely and I would guess a comparison with such would be warranted.
-
The EM problem formulation seems off, usually you would model it as maximizing the ELBO which cancels the evidence term. In the case, it seems to require a normalization constant, which makes it not a EM algorithm. At it just reduces to a ML objective for which the EM proecdure is not needed.
-
The paper misses some important comparisons in my opinion, such as the piecewise normalizing flow [2] and the rejection sampling framework by stimper et al [3]. I would also appreciate a flow matching/diffusion comparison to mitigate my first point.
-
It is unclear how much heavy lifting the cover does. If the density is a GMM, does not the prior already kind of solve it? Wouldnt the algorithm recover the separated modes?
[1] Can Push-forward Generative Models Fit Multimodal Distributions?, Salmona et al
[2] Piecewise Normalizing Flows, Bevin et al
[3] Resampling Base Distributions of Normalizing Flows, Stimper et al
Questions for authors
Basically discuss all the points I outlined in weaknesses. Two more:
-
Of course a larger Lipschitz constant per se is bad, but in what effects does it materialize? Does it affect sampling std, training dynamics?
-
Is there a straightforward way to generalize this method to cond generalization?
8 comments
Response (2/2)
Missing comparisons (W3)
We acknowledge the importance of the flow matching baseline raised by the reviewer and have included flow matching results as an additional baseline, as described in our response above.
We would also like to clarify that Piecewise Normalizing Flows (Bevin et al., 2023) were already included in the original version of our work and are reported in Tables 1 and 2 under the name “k-means PNF.”
Furthermore, the rejection sampling framework of Stimper et al. (2022) was already cited in the original manuscript. As discussed in the Related Work section, rejection sampling from the base distribution comes at the cost of sacrificing its tractability. Our method is designed to preserve the core computational properties of normalizing flows, namely: (i) a simple, tractable base distribution, (ii) efficient and exact likelihood computation, and (iii) efficient one-step sampling. The rejection sampling framework sacrifices all three properties. Flow matching, in turn, retains property (i) but sacrifices property (ii) and, unless an additional flow map is trained, property (iii).
Impact of the contractible cover algorithm (W4)
Gaussian mixture distributions are a standard synthetic benchmark for generative models. They also exhibit non-trivial topology due to their disconnected modes (non-trivial zeroth-order homology). However, in this particular setting, k-means clustering would, in principle, suffice, since its centroids can be interpreted as estimates of the component means of a Gaussian mixture model.
In contrast, the synthetic datasets introduced in our work exhibit non-trivial zeroth- and first-order homology, corresponding to disconnected modes and holes, respectively. The results in Table 1 demonstrate that mixture models achieve better likelihood scores than single-chart models. More importantly, topology-aware partitioning based on contractible covers consistently improves model performance compared to topology-unaware baselines, such as k-means clustering and unregularized mixture models.
We believe these results provide clear evidence of the benefits of topology-aware data partitioning and demonstrate the added value of our cover algorithm beyond conventional clustering approaches.
The role of Lipschitz constants in the evaluation (Q1)
We report the Lipschitz constants of the evaluated methods because prior work (Cornish et al., 2020) developed theoretical results showing that a topological mismatch between the source and target distributions has to manifest as large Lipschitz constants as the model increasingly accurately approximates the target distribution. We therefore use Lipschitz constants as a theory-motivated diagnostic for assessing potential topological mismatch. Nevertheless, as discussed in Section 5.3, large Lipschitz constants are not necessarily indicative of a topological mismatch. They may also arise when the source and target distributions have matching topologies but the target distribution exhibits high curvature. During the rebuttal, we made the Lipschitz constant comparison more principled by normalizing the scores (new Appendix F in the revised pdf) and we compared the topological faithfulness of the learned models by comparing the persistent homology features of their learned density maps (new Appendix I).
Extending the Method to Conditional Generation (Q2)
For a finite set of conditioning values , conditional generation can be viewed as modeling a collection of distributions indexed by . One possible extension would therefore be to partition the training data corresponding to each conditional subpopulation independently and use the union of the resulting labels to index the mixture model. In this way, each conditional distribution would be associated with its own set of mixture components. When is continuous, a possible extension would be to partition the distributions over intervals , potentially under suitable continuity assumptions on the family of conditional distributions . We acknowledge conditional generative modeling as an interesting direction for future research and leave its systematic investigation to future work.
We hope the additional analyses and theoretical results address your questions. We would be happy to clarify anything that remains open. If our responses have resolved your concerns, we would be grateful if you could consider updating your score accordingly.
Response
I thank the authors for their response and additional material. I am keeping my score, as the flow matching comparison shows that my concerns are somehow justified. I remain unconvinced of the significance of the approach given that flow matching performs well out of the box. Note also that 10^4 parameters is extremely low (even for the low dimensions). As you correctly pointed out, flow matching is a kind of normalizing flow and there has been recently a lot of progress in one-step generators with flow matching.
Further, I do not concur with the whole Lipschitz constant as a theoretical measure of the well-posedness. There is likely real effects that a high Lipschitz constants has and showing it is crucial to justify the problem.
EDIT: I thought about it and I like the idea and I am raising my score in light of the nice idea. I would still like a discussion of the practical impact of the Lipschitz constant.
Response (1/2)
We thank the reviewer for their feedback. We are pleased that the reviewer found our method genuinely novel and the results encouraging. We address the reviewer’s concerns and questions below.
Additional flow matching baseline (W1)
We follow the reviewer’s advice and evaluated the performance of the flow matching model and added it as a valuable baseline in the Tables 3, 4, 5, 6, 7, 8, 10. The flow matching model is parametrized by a velocity network with approximately 1.4⋅104 parameters (details in new Appendix E in the revised pdf), compared to small affine model with 1.3⋅104 parameters and big spline model with 6.3⋅104 parameters. The flow matching model performs better than the normalizing flow models on a large subset of datasets, which is expected, especially in the adversarial setup of our work, which is tailored to point out the limitations of normalizing flows. However, on the topologically complex Mickey and Rings datasets, the flow matching model is outperformed even by the relatively small affine CoCoFlow model (Mickey: −1.04±0.06 vs 0.03±0.01; Rings: −2.35±0.02 vs −2.08±0.05). These results highlight that, on datasets with substantial topological structure, even a simple multi-chart model can outperform more expressive architectures that rely on a single chart.
At the same time, we would like to point out that our work, as in the title, is concerned with normalizing flows specifically. The motivation, as framed in the Introduction, was to strengthen the performance of one-step, exact likelihood normalizing flows from standard Gaussian base distribution. Although the flow matching is naturally viewed as a method stemming from the normalizing flows, their computational behavior is distinct from classical normalizing flows. In particular, they have one downside: the log-probability estimates of the flow matching models are expensive to obtain, since they require multi-step integration of the flow, contrary to one-step evaluation in normalizing flows. Moreover, the log-probability returned by the above procedure is approximate in nature because of probability flow ODE discretization error and Jacobian approximation, which exact computation is intractable in high-dimensional cases. Normalizing flows employ constrained architectures which are invertible and have tractable Jacobian determinants that can be evaluated exactly and efficiently.
Clarifications on the training procedure (W2)
We thank the reviewer for pointing out the distinction between our training procedure and standard EM. We agree that our original terminology was imprecise, and we have revised the description to clarify the relationship.
In the special case of α=0 and a trivial cover (i.e., every data point belongs to every cover element), our procedure reduces to generalized EM for a mixture of normalizing flows. In this setting, the E-step computes the standard mixture posterior responsibilities, while the M-step maximizes the expected complete-data log-likelihood. Since the component densities are parameterized by normalizing flows, their parameters are optimized numerically using gradient-based optimization, whereas the mixture weights admit a closed-form update given by the empirical mean of the responsibilities. Provided that the numerical M-step increases its objective, this corresponds to generalized EM.
For a non-trivial cover, however, the membership assignments are restricted to the cover elements containing each data point. Consequently, they no longer correspond to the unrestricted posterior of the mixture model. The resulting alternating updates therefore do not inherit the standard EM guarantee of monotonically increasing the mixture log-likelihood. For α>0, our objective additionally incorporates posterior guidance by balancing mixture likelihood maximization with assignment-weighted conditional likelihood maximization. In particular, at α=1, the objective reduces to weighted likelihood maximization, as employed in Piecewise Normalizing Flows (Bevin et al., 2023). The cover-constrained assignments guide the specialization of individual components to the corresponding regions, without imposing explicit support constraints on the component densities.
We therefore agree that a more precise characterization of our method would be “posterior-guided alternating optimization”, which recovers generalized EM in the limiting case of a trivial cover and α=0. We will update the terminology and mathematical description in the final manuscript accordingly.