- Rating
- 2
- Confidence
- 2
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper considers Picard iterations for the Wasserstein Gradient Flow (WGF) with respect to a smoothed KL. It uses theoretical insights on these iterations to improve generation in the latent space of a pretrained encoder.
Critical strengths
The overall idea is very interesting and the proposed model seems genuinely new.
It combines very interesting theoretical insights on Wasserstein Gradient Flow with a practical algorithm.
The writing is good and nicely motivates everything
Critical weaknesses
- While the model is interesting, I would say that the benchmarking is very much insuffcient. Personally, I do not expect SOTA numbers as long as the idea is new and interesting, but Table 3 is very much unconvincing. There has been so much work on learning Flow Maps and other few-step models and they are nicely summarized in the SuppMat, but the paper mostly fails to compare to them.
An example: Shortcut Models use a different, much higher-dimensional VAE latent space (32×32×4 versus 4×4×8 or 8×8×4), while PC-VAE uses approximately 10× more training sample exposures, excluding VAE pretraining. Consequently, it is unclear whether the reported improvements come from the proposed algorithm or these differences in experimental setup.
-
I am not yet convinced why we need the smoothed KL instead of the actual KL.
-
The paper lacks code, which makes it hard to assess the practical implementation.
MINOR
At several points, the paper is relatively loose about mathematical regularity. While this often improves readability, I believe some assumptions should be made more explicit. For example:
- Lines 131–138: The Wasserstein gradient flow is immediately represented as a pointwise ODE, although this representation requires appropriate differentiability and regularity assumptions on the functional and velocity field.
- Lines 223–225: The unsmoothed score is written in terms of a density, although general measures in need not admit densities.
- Lines 328–336 (Appendix F): The proof of Theorem 1 uses entropy differentiation and integration by parts, whose justification under merely finite initial KL should be clarified.
- Appendix D, lines 1096–1115: The first-variation derivation assumes densities, whereas Theorem 2 includes potentially singular initial measures.
I do not necessarily regard these as errors in the underlying results, but the authors should distinguish formal calculations from statements valid under the advertised minimal assumptions, or briefly explain the necessary approximation arguments.
Questions for authors
Q1 Since the Wasserstein gradient flow of ordinary KL relative to a standard Gaussian is the Ornstein–Uhlenbeck probability flow, whose marginals are explicitly available and whose scores can be learned using denoising score matching, what is the theoretical or practical advantage of considering the smoothed-KL gradient flow instead? In particular, the ordinary KL flow enjoys exponential convergence, while Theorem 1 establishes only an O(1/t) guarantee. Could the authors compare their approach against a trajectory-learning or decoder-adaptation method based on the ordinary KL probability flow?
This is partially addressed this in Appendix E, but I would be interested in empirical evidence for any claimed problems.
Q2 What would be the benchmarking results if we retrained all models in the same pretrained VAE-Latent Space (Flow matching, Consistency Models, Flow Maps, ..) under compute-matched conditions, e.g., approx same FLOPs or sample-budget-matched (training steps × batch size) for identical NN backbone (including any encoder-decoder combination)? Personally, I am less interested in headline numbers than in fair benchmarking, given that the theoretical ideas of the proposed model are already interesting enough. To be clear, I am not expecting large-scale new runs given the duration of the review period, but I think a small benchmarking should be possible for some suitable low-dimensional latent space on selected datasets.
Q3 Could the authors provide an anonymized implementation of PC-VAE during the rebuttal period to facilitate reproducibility and verification of the experimental results?
Q4 Could the authors add an illuminating 2D example? What do the iterations looks like, e.g., for the 'Spirals' or the '2 Moons' datasets?
Q5 Does the method work without an pretrained encoder for simple image datasets, e.g., MNIST? Could one make it work in pixel space? The theoretical formulation does not rely on any decoders after all.
Q6 Could you add some ablations of the pretrained encoder? The one used seems quite powerful.
9 comments
Official Response to Reviewer qTgS - part 2
Q1 (b) [combined with W2] Could the authors compare their approach against a trajectory-learning or decoder-adaptation method based on the ordinary KL probability flow?
We performed the following experiment on the CelebA-HQ dataset with 256x256 resolution, where we first train a conditional flow model based on the rectified flow path (which is well known to be a reparameterization of the OU path) on the latent space of the pre-trained autoencoder, then we numerically simulate the reversal of this flow to obtain the Gaussianized latent, from which we adapt the decoder. We test two variants: one where the flow reversal is initialized from the exact latent (), and one where the latent is first noised to where ), and the flow reversal is initialized from the noised samples, starting from . The results are summarized in Table 12 in Appendix I.7.1, and we report it below:
| Method | Generation FID ↓ | Reconstruction FID ↓ |
|---|---|---|
| Rectified flow inversion + decoder fine-tuning (t₀ = 1) | 67.51 | 1.11 |
| Rectified flow inversion + decoder fine-tuning (t₀ = 0.9) | 12.16 | 10.97 |
| PC-VAE (Ours) | 9.95 | 2.04 |
We observe that for , while the reconstruction is good, the generative quality is really bad ( > 60 FID). Conversely, for the generation is much better, but reconstruction quality drops by a lot. Intuitively, our reasoning for this behaviour is as follows: near , the score/velocity field is much more difficult to learn. Thus, when we start from and run a few Euler steps for the reverse flow, errors can accumulate early on, and this can result in off-policy samples for later time-steps, leading to more errors and eventually to a distribution that is not well Gaussianized. We empirically tested this by measuring various latent metrics (see Table 13, Appendix I.7), and the deviation from was much higher for than for .
Q2. (combined with W1) While the model is interesting, I would say that the benchmarking is very much insuffcient… What would be the benchmarking results if we retrained all models in the same pretrained VAE-Latent Space (Flow matching, Consistency Models, Flow Maps, ..) under compute-matched conditions, e.g., approx same FLOPs or sample-budget-matched (training steps × batch size) for identical NN backbone (including any encoder-decoder combination)?
Thank you for the suggestion. To ensure fair benchmarking, we have additionally implemented Shortcut Model, MeanFlow and rectified flow-based conditional Flow Matching on the same pre-trained VAE latent space, and under similar compute budgets, for CelebA-HQ with 256x256 resolution. The results are present in Table 11 in the newly added Appendix I.7, and we summarize them below.
| Method | NFE | FID ↓ |
|---|---|---|
| PC-VAE (ε = 0.05, T = 5) | 1 | 9.95 |
| Flow Matching | 50 | 18.25 |
| Flow Matching | 100 | 17.25 |
| Shortcut | 1 | 26.77 |
| Shortcut | 2 | 24.78 |
| Shortcut | 4 | 24.93 |
| Shortcut | 8 | 24.40 |
| MeanFlow | 1 | 25.46 |
| MeanFlow | 2 | 22.25 |
| MeanFlow | 4 | 17.59 |
| MeanFlow | 8 | 10.68 |
In the compared setting, we outperform flow-matching and the other one step generative baselines. However, we also note that for the baselines we used the hyper-parameters from their original implementations, and we didn’t specifically tune them.
Q3 (combined with W3) The paper lacks code, which makes it hard to assess the practical implementation. Could the authors provide an anonymized implementation of PC-VAE during the rebuttal period to facilitate reproducibility and verification of the experimental results?
While we are in the process of trying to make the code public, currently due to IP restrictions, we are unable to share the original implementation of PC-VAE. In the meantime, we provide a minimal implementation of PC-VAE for MNIST (in pixel space) which illustrates the core algorithm. Link: https://anonymous.4open.science/r/pcvae-pixel-mnist-5A70/
Q4. Could the authors add an illuminating 2D example? What do the iterations looks like, e.g., for the 'Spirals' or the '2 Moons' datasets?
We have now added Figures 3, 4 and 5 (Appendix I.2), which illustrate the learnt WGF and the Picard iterations for the two datasets, across different t.
Official Response to Reviewer qTgS - part 3
Q5. Does the method work without a pretrained encoder for simple image datasets, e.g., MNIST? Could one make it work in pixel space? The theoretical formulation does not rely on any decoders after all.
Theoretically yes, PC-VAE could work directly in pixel space. This would involve learning the smoothed KL flow, with the initial distribution as the image distribution. Once the flow is learnt, a decoder is still required to reconstruct from the end-point of the flow (i.e., should match with ). Practically, it is difficult to implement PC-VAE directly in pixel space when the dimensions are large (e.g., 256x256x3), as we would require significantly more compute. Additionally, from Theorem 1 the smoothed KL WGF requires that the initial distribution has finite KL divergence with respect to in order for it to converge, and since the image distribution is typically supported on a much lower dimensional manifold, this condition is not satisfied. For this reason, we believe that PC-VAE will perform better when implemented in the latent space of a pre-trained autoencoder, since presumably the push-forward of the data distribution under the encoder map has full support, and possibly a finite KL divergence with respect to .
Nevertheless, we demonstrate that PC-VAE is capable of working directly in the pixel space for the MNIST dataset. In order to make it work, we observed that it is necessary to add some noise to the images before performing the algorithm in order to “lift” the data manifold to the ambient dimension, i.e., the initial distribution is not the image distribution, but rather a noised version of it. More implementation details and figures are present in Appendix I.3.
Q6. Could you add some ablations of the pretrained encoder? The one used seems quite powerful.
We are currently running ablations for the pre-trained encoder, however due to compute related issues it is taking longer than expected. We will update you as soon as the ablations are done, before the deadline.
Official Response to Reviewer qTgS - part 1
Dear reviewer qTgS,
Thank you for highlighting that our work has very interesting ideas and theoretical insights. We are glad that you found the underlying method novel, and that our presentation is well motivated. We have incorporated your suggestions in the revised draft (new additions are colored green), and we address your questions and concerns below.
Weaknesses:
Thank you for the feedback. We have addressed the specific points raised above in the revised draft. In particular,
Questions
There are a few reasons why we consider the smoothed KL instead of the ordinary KL flow. Firstly, learning the ordinary KL flow using Picard iteration would involve estimating the exact score ∇logpXtn along the current trajectory Xtn , which is computationally much harder than estimating the smoothed score ∇logpXtn,ϵ required by the smoothed KL flow (which can be done efficiently using DSM). Therefore, if we decide that the underlying flow is to be learnt using Picard iteration, then the smoothed KL flow is practically the better choice.
Secondly, as you have rightly pointed out, the ordinary KL flow has explicitly available marginals whose scores can be learnt using DSM. So, a natural approach would be to learn the scores of the ordinary KL flow, and use it to learn a flow-map from data to noise (e.g., via distillation), following which the decoder can be adapted. However, such an approach would either involve evaluating the learnt scores on out-of-distribution samples (as discussed in Appendix E) or numerically simulating trajectories, neither of which is attractive. Practically, PC-VAE with the smoothed KL functional has an advantage that it is entirely on-policy; the smoothed scores are learnt from the current trajectory, and during the trajectory update phase the frozen score network is also evaluated on samples from the current trajectory.