acceptodds
Under review as a conference paper at ICLR 2027

Generative Auto Encoders via Picard Iterated Wasserstein Gradient Flows

Abstract

Variational Auto-Encoders are trained with the goal of reconstructing the inputs from the latents accurately, and they also attempt to match the latent distribution to a desired prior (typically standard Gaussian). Due to the tradeoff between reconstruction and prior regularization in ELBO-based training, the latent distribution is far from the desired prior, which results in poor quality of VAEs when used as generative models. Motivated by the problem of shaping a latent distribution to the Gaussian, we introduce the smoothed KL divergence, a geodesically non-convex functional on the space of probability distributions and consider its Wasserstein Gradient Flow (WGF). This WGF requires only the score function of the smoothed distribution and is easily learned via the popular denoising score matching. We establish theoretically that it converges to the target distribution (Gaussian) via a Lyapunov decay argument, bypassing Polyak-Lojaciewicz type conditions usually used in this context. We apply the smoothed KL WGF to derive a novel generative modeling algorithm, based on autoencoders. Our algorithm introduces a time-parameterized neural network in the latent space to represent trajectories, and during training we iteratively update the entire family of trajectories using Picard's iteration, with the goal of eventually aligning the trajectories to the WGF. Simultaneously, we adapt the decoder to reconstruct from the end-points of the trajectories using autoencoder reconstruction losses. We establish theoretical guarantees for convergence of the Picard iterates to the chosen WGF, as well as convergence of the WGF to Gaussian, under mild assumptions on the initial latent distribution. We empirically establish the efficacy of our method on the unconditional datasets CelebA-HQ, LSUN Church and LSUN Bedroom.

Primary area:
generative models
Keywords:
Variational Autoencoders, Picard's Iteration, Generative Modeling

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_qTgSOpenReview ↗
Rating
2
Confidence
2
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

The paper considers Picard iterations for the Wasserstein Gradient Flow (WGF) with respect to a smoothed KL. It uses theoretical insights on these iterations to improve generation in the latent space of a pretrained encoder.

Critical strengths

The overall idea is very interesting and the proposed model seems genuinely new.

It combines very interesting theoretical insights on Wasserstein Gradient Flow with a practical algorithm.

The writing is good and nicely motivates everything

Critical weaknesses
  1. While the model is interesting, I would say that the benchmarking is very much insuffcient. Personally, I do not expect SOTA numbers as long as the idea is new and interesting, but Table 3 is very much unconvincing. There has been so much work on learning Flow Maps and other few-step models and they are nicely summarized in the SuppMat, but the paper mostly fails to compare to them.

An example: Shortcut Models use a different, much higher-dimensional VAE latent space (32×32×4 versus 4×4×8 or 8×8×4), while PC-VAE uses approximately 10× more training sample exposures, excluding VAE pretraining. Consequently, it is unclear whether the reported improvements come from the proposed algorithm or these differences in experimental setup.

  1. I am not yet convinced why we need the smoothed KL instead of the actual KL.

  2. The paper lacks code, which makes it hard to assess the practical implementation.

MINOR

At several points, the paper is relatively loose about mathematical regularity. While this often improves readability, I believe some assumptions should be made more explicit. For example:

  • Lines 131–138: The Wasserstein gradient flow is immediately represented as a pointwise ODE, although this representation requires appropriate differentiability and regularity assumptions on the functional and velocity field.
  • Lines 223–225: The unsmoothed score is written in terms of a density, although general measures in P2\mathcal P_2 need not admit densities.
  • Lines 328–336 (Appendix F): The proof of Theorem 1 uses entropy differentiation and integration by parts, whose justification under merely finite initial KL should be clarified.
  • Appendix D, lines 1096–1115: The first-variation derivation assumes densities, whereas Theorem 2 includes potentially singular initial measures.

I do not necessarily regard these as errors in the underlying results, but the authors should distinguish formal calculations from statements valid under the advertised minimal assumptions, or briefly explain the necessary approximation arguments.

Questions for authors

Q1 Since the Wasserstein gradient flow of ordinary KL relative to a standard Gaussian is the Ornstein–Uhlenbeck probability flow, whose marginals are explicitly available and whose scores can be learned using denoising score matching, what is the theoretical or practical advantage of considering the smoothed-KL gradient flow instead? In particular, the ordinary KL flow enjoys exponential convergence, while Theorem 1 establishes only an O(1/t) guarantee. Could the authors compare their approach against a trajectory-learning or decoder-adaptation method based on the ordinary KL probability flow?

This is partially addressed this in Appendix E, but I would be interested in empirical evidence for any claimed problems.

Q2 What would be the benchmarking results if we retrained all models in the same pretrained VAE-Latent Space (Flow matching, Consistency Models, Flow Maps, ..) under compute-matched conditions, e.g., approx same FLOPs or sample-budget-matched (training steps × batch size) for identical NN backbone (including any encoder-decoder combination)? Personally, I am less interested in headline numbers than in fair benchmarking, given that the theoretical ideas of the proposed model are already interesting enough. To be clear, I am not expecting large-scale new runs given the duration of the review period, but I think a small benchmarking should be possible for some suitable low-dimensional latent space on selected datasets.

Q3 Could the authors provide an anonymized implementation of PC-VAE during the rebuttal period to facilitate reproducibility and verification of the experimental results?

Q4 Could the authors add an illuminating 2D example? What do the iterations looks like, e.g., for the 'Spirals' or the '2 Moons' datasets?

Q5 Does the method work without an pretrained encoder for simple image datasets, e.g., MNIST? Could one make it work in pixel space? The theoretical formulation does not rely on any decoders after all.

Q6 Could you add some ablations of the pretrained encoder? The one used seems quite powerful.

9 comments
AuthorsOpenReview ↗

Official Response to Reviewer qTgS - part 1

Dear reviewer qTgS,

Thank you for highlighting that our work has very interesting ideas and theoretical insights. We are glad that you found the underlying method novel, and that our presentation is well motivated. We have incorporated your suggestions in the revised draft (new additions are colored green), and we address your questions and concerns below.

Weaknesses:

W4. At several points, the paper is relatively loose about mathematical regularity. While this often improves readability, I believe some assumptions should be made more explicit...

Thank you for the feedback. We have addressed the specific points raised above in the revised draft. In particular,

  • We have now added a phrase “assuming F\mathcal{F} satisfies standard regularity conditions” in Section 3, along with a reference to Gradient flows: in metric spaces and in the space of probability measures, (Springer, 2005) by Ambrosio et al.
  • In the context of Section 4.2, ptp_t corresponds to the marginal at time t of the ordinary KL WGF (equivalently the OU marginal at time t), hence they have smooth and strictly positive densities for all t > 0 (hence their unsmoothed scores exist). We have elaborated on this in Appendix B.1.
  • We have now added an alternate and more rigorous proof in Appendix F.1 for Theorem 1 under the minimal assumption that KL(p0∥q)<∞KL(p_0 \| q) < \infty. Thus, theorem 1 remains unaffected.
  • We rewrote the first-variation derivation in Appendix D using general measures pp and perturbation δp\delta p (without assuming densities), since the proof did not actually rely on the densities existing. Theorem 2 remains unaffected.

Questions

Q1 (a) [combined with W2] I am not yet convinced why we need the smoothed KL instead of the actual KL. Since the WGF of ordinary KL relative to a standard Gaussian is the OU probability flow, whose marginals are explicitly available and whose scores can be learned using denoising score matching, what is the theoretical or practical advantage of considering the smoothed-KL gradient flow instead?

There are a few reasons why we consider the smoothed KL instead of the ordinary KL flow. Firstly, learning the ordinary KL flow using Picard iteration would involve estimating the exact score ∇log⁡pXtn\nabla \log p_{X^n_t} along the current trajectory XtnX^n_t , which is computationally much harder than estimating the smoothed score ∇log⁡pXtn,ϵ\nabla \log p_{X^n_t, \epsilon} required by the smoothed KL flow (which can be done efficiently using DSM). Therefore, if we decide that the underlying flow is to be learnt using Picard iteration, then the smoothed KL flow is practically the better choice.

Secondly, as you have rightly pointed out, the ordinary KL flow has explicitly available marginals whose scores can be learnt using DSM. So, a natural approach would be to learn the scores of the ordinary KL flow, and use it to learn a flow-map from data to noise (e.g., via distillation), following which the decoder can be adapted. However, such an approach would either involve evaluating the learnt scores on out-of-distribution samples (as discussed in Appendix E) or numerically simulating trajectories, neither of which is attractive. Practically, PC-VAE with the smoothed KL functional has an advantage that it is entirely on-policy; the smoothed scores are learnt from the current trajectory, and during the trajectory update phase the frozen score network is also evaluated on samples from the current trajectory.

AuthorsOpenReview ↗

Official Response to Reviewer qTgS - part 2

Q1 (b) [combined with W2] Could the authors compare their approach against a trajectory-learning or decoder-adaptation method based on the ordinary KL probability flow?

We performed the following experiment on the CelebA-HQ dataset with 256x256 resolution, where we first train a conditional flow model based on the rectified flow path (which is well known to be a reparameterization of the OU path) on the latent space of the pre-trained autoencoder, then we numerically simulate the reversal of this flow to obtain the Gaussianized latent, from which we adapt the decoder. We test two variants: one where the flow reversal is initialized from the exact latent (t0=1t_0 = 1), and one where the latent xx is first noised to 0.9x+0.1z0.9x + 0.1z where z∼N(0,I)z \sim \mathcal{N}(0,I)), and the flow reversal is initialized from the noised samples, starting from t0=0.9t_0 = 0.9. The results are summarized in Table 12 in Appendix I.7.1, and we report it below:

MethodGeneration FID ↓Reconstruction FID ↓
Rectified flow inversion + decoder fine-tuning (t₀ = 1)67.511.11
Rectified flow inversion + decoder fine-tuning (t₀ = 0.9)12.1610.97
PC-VAE (Ours)9.952.04

We observe that for t0=1t_0=1, while the reconstruction is good, the generative quality is really bad ( > 60 FID). Conversely, for t0=0.9t_0=0.9 the generation is much better, but reconstruction quality drops by a lot. Intuitively, our reasoning for this behaviour is as follows: near t0=1t_0=1, the score/velocity field is much more difficult to learn. Thus, when we start from t0=1t_0=1 and run a few Euler steps for the reverse flow, errors can accumulate early on, and this can result in off-policy samples for later time-steps, leading to more errors and eventually to a distribution that is not well Gaussianized. We empirically tested this by measuring various latent metrics (see Table 13, Appendix I.7), and the deviation from N(0,I)\mathcal{N}(0,I) was much higher for t0=1t_0=1 than for t0=0.9t_0=0.9.

Q2. (combined with W1) While the model is interesting, I would say that the benchmarking is very much insuffcient… What would be the benchmarking results if we retrained all models in the same pretrained VAE-Latent Space (Flow matching, Consistency Models, Flow Maps, ..) under compute-matched conditions, e.g., approx same FLOPs or sample-budget-matched (training steps × batch size) for identical NN backbone (including any encoder-decoder combination)?

Thank you for the suggestion. To ensure fair benchmarking, we have additionally implemented Shortcut Model, MeanFlow and rectified flow-based conditional Flow Matching on the same pre-trained VAE latent space, and under similar compute budgets, for CelebA-HQ with 256x256 resolution. The results are present in Table 11 in the newly added Appendix I.7, and we summarize them below.

MethodNFEFID ↓
PC-VAE (ε = 0.05, T = 5)19.95
Flow Matching5018.25
Flow Matching10017.25
Shortcut126.77
Shortcut224.78
Shortcut424.93
Shortcut824.40
MeanFlow125.46
MeanFlow222.25
MeanFlow417.59
MeanFlow810.68

In the compared setting, we outperform flow-matching and the other one step generative baselines. However, we also note that for the baselines we used the hyper-parameters from their original implementations, and we didn’t specifically tune them.

Q3 (combined with W3) The paper lacks code, which makes it hard to assess the practical implementation. Could the authors provide an anonymized implementation of PC-VAE during the rebuttal period to facilitate reproducibility and verification of the experimental results?

While we are in the process of trying to make the code public, currently due to IP restrictions, we are unable to share the original implementation of PC-VAE. In the meantime, we provide a minimal implementation of PC-VAE for MNIST (in pixel space) which illustrates the core algorithm. Link: https://anonymous.4open.science/r/pcvae-pixel-mnist-5A70/

Q4. Could the authors add an illuminating 2D example? What do the iterations looks like, e.g., for the 'Spirals' or the '2 Moons' datasets?

We have now added Figures 3, 4 and 5 (Appendix I.2), which illustrate the learnt WGF and the Picard iterations for the two datasets, across different t.

AuthorsOpenReview ↗

Official Response to Reviewer qTgS - part 3

Q5. Does the method work without a pretrained encoder for simple image datasets, e.g., MNIST? Could one make it work in pixel space? The theoretical formulation does not rely on any decoders after all.

Theoretically yes, PC-VAE could work directly in pixel space. This would involve learning the smoothed KL flow, with the initial distribution as the image distribution. Once the flow is learnt, a decoder is still required to reconstruct from the end-point of the flow (i.e., D(Ψ(X0,T))D( \Psi(X_0, T) ) should match with X0X_0). Practically, it is difficult to implement PC-VAE directly in pixel space when the dimensions are large (e.g., 256x256x3), as we would require significantly more compute. Additionally, from Theorem 1 the smoothed KL WGF requires that the initial distribution has finite KL divergence with respect to N(0,I)\mathcal{N}(0,I) in order for it to converge, and since the image distribution is typically supported on a much lower dimensional manifold, this condition is not satisfied. For this reason, we believe that PC-VAE will perform better when implemented in the latent space of a pre-trained autoencoder, since presumably the push-forward of the data distribution under the encoder map has full support, and possibly a finite KL divergence with respect to N(0,I)\mathcal{N}(0,I).

Nevertheless, we demonstrate that PC-VAE is capable of working directly in the pixel space for the MNIST dataset. In order to make it work, we observed that it is necessary to add some noise to the images before performing the algorithm in order to “lift” the data manifold to the ambient dimension, i.e., the initial distribution is not the image distribution, but rather a noised version of it. More implementation details and figures are present in Appendix I.3.

Q6. Could you add some ablations of the pretrained encoder? The one used seems quite powerful.

We are currently running ablations for the pre-trained encoder, however due to compute related issues it is taking longer than expected. We will update you as soon as the ablations are done, before the deadline.

Reviewer_wA5COpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
1
Soundness
3
Read review
Paper summary

The authors propose a new two stage process to fixe the non Gaussian prior which emerges from a trained VAE. After a vanilla VAE training, the aim is to transport the real data driven prior (given by the embeddings) to the expected VAE model priori: the standard normal distribution. This transport is done by learning a continuous trajectory between the two distributions. Unlike latent diffusion models, which learn the vector field step by step over time, PC-VAE learns the entire set of trajectories in latent space using the Picard iteration (via a more stable semi-implicit version). The decoder is therefore adapted to this new prior so that the generation summarises in decoding a standard normal sample. Theoretical guaranties are proved to support the approach.

Critical strengths

1- the theoretical guaranties are very interesting and makes the method well-founded. 2- The one-step generation process is valuable.

Critical weaknesses

1- the paper is hard to read for non experts. The concepts are introduced in section 3 without any context when mentioned earlier. This makes the reader go back and forth all along the review. 2- Although the one-step generation is valuable, the training looks quite computationally heavy. The question of scalability arises. 3- The comparison is reduced to specific models. It would have been interesting to see the comparison with classical VAE-like models in particulars the ones trying to explicit the prior as a posterior (like here) either within a single training (VAE with VAMP prior -https://arxiv.org/abs/1705.07120, VAE with Riemannian structure in the latent space - https://arxiv.org/pdf/2002.05227) or as a second step (https://arxiv.org/abs/2209.07370).

Questions for authors

1- Could you isolate the contribution of the semi-implicit Picard iteration in a ablation study? For example, comparing the method with a standard Picard iteration to a simpler latent Gaussianization approach (using optimal transport?) and the same decoder adaptation? 2- How does the method scale with data dimension, latent space dimension? 3- Can you add at least 2 more "standard" VAE - like methods to compare?

1 comment
AuthorsOpenReview ↗

Official Response to Reviewer wA5C

Dear Reviewer wA5C,

Thank you for highlighting that our work has very interesting theoretical guarantees, making our one-step generation method well founded. We have incorporated your suggestions in the revised draft (new additions are colored green), and we address your questions and concerns below.

Weaknesses

W1. The paper is hard to read for non experts. The concepts are introduced in section 3 without any context when mentioned earlier.

Thank you for your feedback regarding the readability. We have revised the introduction to improve the exposition.

W2. Although the one-step generation is valuable, the training looks quite computationally heavy.

To contextualize the training cost for our runs, we compare our method with other baselines using similar training compute (and performed on the same pre-trained VAE) on CelebA-HQ with 256x256 resolution, and we report the results below. More details can be found in Appendix I.7.1.

MethodNFEFID ↓
PC-VAE (ε = 0.05, T = 5)19.95
Flow Matching5018.25
Flow Matching10017.25
Shortcut126.77
Shortcut224.78
Shortcut424.93
Shortcut824.40
MeanFlow125.46
MeanFlow222.25
MeanFlow417.59
MeanFlow810.68

In this matched setting, we observe that PC-VAE performs better than the baselines. However, we also note that we used the original hyperparameters for the baselines; we did not specifically tune them.

Questions

Q1. Could you isolate the contribution of the semi-implicit Picard iteration in a ablation study? For example, comparing the method with a standard Picard iteration to a simpler latent Gaussianization approach (using optimal transport?) and the same decoder adaptation?

Thank you for the suggestion. From our understanding, the reviewer’s question is whether we can use a different Gaussianization approach for latents instead of our PC-VAE method. To test this, we performed an experiment where we first train a conditional flow model based on the rectified flow path in the latent space of the pre-trained autoencoder, and then we transport the latent space by numerically simulating the reversal of the flow model, following which we do the same decoder adaptation. We test two variants of this: one where the flow reversal is initialized from the exact latent (t0=1t_0 = 1), and one where the latent xx is first noised to 0.9x+0.1z0.9x + 0.1z where z∼N(0,I)z \sim \mathcal{N}(0,I)), and the flow reversal is initialized from the noised samples, starting from t0=0.9t_0 = 0.9. We have now added this comparison in Table 12 of Appendix I.7.1, and we summarize the results below (on CelebA-HQ with 256x256 resolution).

MethodGeneration FID ↓
PC-VAE9.95
Rectified flow inversion + decoder fine-tuning (t0=1t_0=1)67.51
Rectified flow inversion + decoder fine-tuning (t0=0.9t_0=0.9)12.16

Q2. How does the method scale with data dimension, latent space dimension?

We have an ablation on latent size in Table 9 (Appendix I.5), which we report here for reference.

Latent shapeCelebA-HQ FID ↓LSUN Church FID ↓LSUN Bedroom FID ↓
4 × 4 × 810.686.49–
8 × 8 × 418.568.688.03
16 × 16 × 424.8712.9011.04

We do see a degradation in generative quality as the latent size is increased. However, we note that this is under a matched setting where we maintain the same neural network size as well as training iterations across different latent sizes.

Due to limited compute budget, we were unable to perform experiments to scale our method to larger data dimensions over the rebuttal period.

Q3. (combined with W3) The comparison is reduced to specific models. It would have been interesting to see the comparison with classical VAE-like methods... Can you add at least 2 more "standard" VAE - like methods to compare?

We have now added a comparison of our method with WAE-MMD and VampPrior, in Table 14 (Appendix I.7.2), and we summarize the results below (on CelebA-HQ with 64x64 resolution).

MethodGeneration FID ↓Reconstruction FID ↓
VAE52.251.54
WAE-MMD13.811.15
VampPrior14.010.98
PC-VAE after 5 iterations12.118.73
PC-VAE after 10 iterations6.773.08
PC-VAE after 20 iterations5.471.68
PC-VAE after 20 iterations + 10k final decoder steps5.671.40
Reviewer_HPzaOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes the Picard-Corrected VAE (PC-VAE) to correct the mismatch between the aggregated latent distribution of a pretrained VAE and a standard Gaussian prior. The core idea is to define the process of shifting the encoded latent distribution toward a Gaussian distribution as a Wasserstein gradient flow (WGF) of the smoothed KL divergence. Instead of directly integrating this flow over time, the entire trajectory is parameterized using a time-dependent neural network and then trained iteratively via a semi-implicit Picard iteration. The required smoothed score is estimated using denoising score matching (DSM).
Specifically, the original encoder EE is fixed, and a trajectory map PsiθPsi_\theta is applied to the encoder’s output. Therefore, the effective encoder after correction can be expressed as

.$$ The decoder $D_\omega$ is adapted to reconstruct the original data from this terminal latent. Consequently, while $E_{\mathrm{eff}}$ is used for reconstruction, one-step generation is possible during generation because a Gaussian sample $z\sim N(0,I)$ is directly fed into the adapted decoder, eliminating the need to evaluate the score network or trajectory map. Theoretically, the authors show that (i) the smoothed-KL WGF converges to the standard Gaussian and that the corresponding functional decreases at a rate of $O(1/t)$, and (ii) semi-implicit Picard iterates converge $L_2$-uniformly to the WGF solution over a finite time interval. Experiments were conducted on 256×256 unconditional generation datasets, namely CelebA-HQ, LSUN Bedroom, and LSUN Church, where the PC-VAE achieved better FID than pretrained VAEs and one- or four-step Shortcut Models, demonstrating the feasibility of one-step generation.
Critical strengths
  1. This paper presents a mathematically consistent latent correction framework. The overall construction is principled: it derives the WGF from the smoothed KL functional, enables the estimation of the required score via DSM using Gaussian smoothing, and approximates this using a semi-implicit Picard iteration. In particular, the analysis that circumvents the problem of the uniform PL inequality not holding for the smoothed KL itself by utilizing the unsmoothed KL Lyapunov functional represents an interesting theoretical contribution.

  2. The theoretical analysis of the semi-implicit Picard iteration is well-aligned with the motivation of the method. The authors discuss the long-horizon instability of the standard Picard iteration and propose a more stable semi-implicit update. They also present a result showing that the iterates converge to a WGF solution under the relatively weak condition of finite second moments.

  3. The idea of separating training-time correction from inference-time generation is compelling. While score and trajectory networks are required during training, generation only requires a single application of the adapted decoder to a Gaussian sample. Thus, one-step generation is achieved by utilizing complex latent transport at training-time.

  4. The experimental results are promising from the perspective of one-step generation. PC-VAE achieves significantly lower FID than one-step Shortcut Models on all three datasets and even outperforms one-step Consistency Distillation on the LSUN Bedroom dataset.

Critical weaknesses
  1. The novelty relative to two-stage autoencoders and latent score/diffusion models is not sufficiently established. The primary concern is that, despite the fact that the overall generative framework is quite closely related to existing two-stage/probabilistic autoencoders and latent score/diffusion models, the relationship with these prior studies has not been sufficiently discussed or experimentally distinguished. As such, this research framework is closely related to the Two-Stage VAE by Dai and Wipf (2019), the Probabilistic Autoencoder (PAE) by Böhm and Seljak (2022), “Diffusion Priors in VAEs” by Wehenkel and Louppe (2021)’s “Diffusion Priors in VAEs,” and, in particular, “Latent Score-based Generative Model” (LSGM) by Vahdat et al. (2021) [1–4]. Although these studies employ different methods, they all address the mismatch between the latent distribution generated by the autoencoder/VAE and a simple prior through a separate, flexible latent generative model. In particular, LSGM directly trains a score-based generative model in the VAE latent space. PC-VAE also trains a Gaussian-smoothed latent score using standard denoising score matching, and this paper explicitly states that this is a core tool in standard diffusion model training. Therefore, the high-level construction of PC-VAE itself, that is, obtaining a reconstruction-oriented latent representation and then resolving the mismatch between the encoded latent distribution and the tractable Gaussian prior through flexible latent generative modeling, can hardly be considered novel. As I understand it, the substantive distinction of PC-VAE is more narrow. While PAE or latent diffusion/score models typically use a separate generative process during generation to transform a simple base distribution to the learned latent distribution, the PC-VAE learns a transport in the opposite direction, i.e.,
.$$ By performing this via semi-implicit Picard iteration and adapting the decoder to the terminal latents, it eliminates inference-time transport. Therefore, I believe the true novelty of this paper lies not so much in latent distribution modeling itself, but rather in the specific training construction. which involves learning the smoothed KL WGF via semi-implicit Picard iteration and combining it with decoder adaptation to eliminate inference-time latent transport, and the theoretical analysis thereof. However, the current experiments lack a controlled baseline that trains a conventional latent diffusion/score prior on the same pretrained autoencoder. Consequently, it is difficult to determine whether the performance improvement stems from the Picard-iterated WGF or from the more general approach of combining well-known expressive latent modeling with decoder retraining. 2. The ablations do not isolate the key methodological contribution The current ablations analyze the smoothing level $\epsilon$, terminal time $T$, and latent dimensionality, but they do not isolate the effects of the Picard iteration, the WGF objective, and decoder adaptation individually. In particular, I believe the following four controlled comparisons are necessary: 1) decoder adaptation only, 2) conventional latent score/diffusion model, 3) conventional latent score/diffusion model + the same decoder adaptation, 4) proposed Picard-WGF + decoder adaptation. Without these comparisons, it is difficult to determine the empirical necessity of the core algorithmic contribution—trajectory-space Picard learning. 3. It is unclear why the original encoder must be frozen rather than directly adapted PC-VAE freezes the original encoder $E$ and adds a learned deterministic transport $\Psi_T$ after it. Therefore, the effective encoder of the final reconstruction model is effectively $$E_{\mathrm{eff}}=\Psi_T\circ E ,$$ a new encoder. The paper also uses precisely this composition during reconstruction. From this perspective, a simpler baseline naturally comes to mind. If what PC-VAE ultimately aims for is $$E_{\mathrm{eff}}\sharp p_{\mathrm{data}} \approx N(0,I)$$ while ensuring that $D_\omega\circ E_{\mathrm{eff}}$ preserves the reconstruction, it is unclear why we must freeze the pretrained $E$ and train a separate time-dependent trajectory network. For example, it seems necessary to compare this approach with a simpler baseline that applies regularization to align the aggregated latent distribution with a Gaussian distribution while directly fine-tuning the encoder and decoder. If such a baseline achieves a similar generation/reconstruction trade-off, the practical benefits provided by separate WGF trajectory learning may be diminished. 4. The prior matching of the learned effective encoder is central to the method but is not sufficiently established at finite training time. I believe this point is particularly important for the one-step generation mechanism of PC-VAE. During training, the decoder is trained to reconstruct $$z_T=\Psi_T(E(x)) .$$ Therefore, the decoder’s training latent distribution is $$p_T=(\Psi_T\circ E)\sharp p_{\mathrm{data}} .$$ In contrast, during generation, the trajectory map is not used. Instead, $$z\sim q=N(0,I)$$ is directly fed to the decoder. Therefore, for PC-VAE’s one-step generation to succeed, $$p_T\approx q$$ must hold with sufficient strength in the actual learned finite-$T$ model. Otherwise, a residual mismatch exists between the decoder’s training distribution $p_T$ and the generation distribution $q$, which is essentially the same type of problem as the VAE prior mismatch that this paper aims to solve. Theorem 1 shows that, for an ideal WGF, $$D_{\mathrm{KL}}(p_{t,\epsilon}\Vert q) \le \frac{D_{\mathrm{KL}}(p_0\Vert q)}{2t}$$ and ultimately proves that $W_2(p_t,q)\to 0$. Theorem 2 shows that exact semi-implicit Picard iterates converge to an ideal WGF trajectory within a finite time interval. However, even when these two results are combined, no direct guarantee regarding the prior mismatch in actual finite-capacity neural PC-VAEs is obtained. Theorem 1 is an asymptotic result for the ideal WGF, and the explicit $O(1/t)$ rate applies to the smoothed KL. Theorem 2 also assumes exact Picard updates. In actual learned models, there are finite Picard iterations, score estimation error, trajectory approximation error, and SGD optimization error. Conceptually, what is actually required is $$W_2(\hat{p}_T,q) \le W_2(\hat{p}_T, p_T^{\mathrm{WGF}}) + W_2(p_T^{\mathrm{WGF}}, q) .$$ That is, a result that simultaneously controls the learning/Picard approximation error and the finite-time Gaussianization error. Current theory does not directly provide this finite-training guarantee for the actual learned $\hat{p}_T$. 5. The existing latent-distribution diagnostics are relevant but insufficient to establish the required prior matching. The authors acknowledge this issue in Appendix H.4 and provide additional diagnostics to verify the Gaussianity of the terminal latent distribution. Specifically, they estimate the relative Fisher information between the smoothed terminal distribution and $N(0,I)$ for $$x_T^n=\Psi_\theta(E(x),T)$$ along the Picard iteration. While this is relevant evidence, however, in this diagnostic, the unknown terminal score is directly approximated by the score network $s_\phi(\cdot,T)$ learned from the PC-VAE. Therefore, since the estimator used to learn the correction dynamics is also used for the testing Gaussianity, it is difficult to consider this as a completely independent validation. It would be more convincing to provide distributional metrics independent of the score network—such as sliced Wasserstein distance, MMD, mean/covariance mismatch, or other Gaussianity diagnostics—for the corrected latents of the held-out data. In particular, if we could show both how $$p_T^{(n)}\rightarrow N(0,I)$$ it would be possible to directly verify the mechanism claimed by the paper. 6. There is a nontrivial gap between the theoretical guarantees and the actual neural training algorithm. While Theorem 2 concerns exact population scores and exact Picard updates, the actual algorithm uses finite-capacity score/trajectory networks and SGD. The paper itself explicitly states the condition “if we are able to update the trajectories perfectly using Picard’s iteration” when explaining the algorithmic implications of the theorem. Therefore, a robustness analysis of how score-estimation error and trajectory-fitting error propagate through Picard iterations would significantly strengthen the theoretical claim. 7. The contribution of decoder adaptation to the improvement in generation quality is unclear. In PC-VAE, not only latent transport but also the decoder is retrained using L1, LPIPS, and adversarial objectives. Therefore, based on the current results alone, it is unclear how much of the improvement in generation FID stems from latent Gaussianization/Picard-WGF and how much from decoder adaptation. In particular, since the naive Gaussian generation FID of a pretrained VAE is very poor, the mere fact that there is a significant improvement over the pretrained VAE is difficult to view as a strong baseline that proves the necessity of Picard-WGF. 8. Theoretically, an ideal WGF converges to a Gaussian prior as $T$ increases. Therefore, according to the motivation behind our method, a larger $T$ should be advantageous, at least from the perspective of latent-prior mismatch. However, as shown in Table 9, as $T$ increases from 5 to 7.5 and 10, the generation FID consistently worsens across all three datasets. I believe this phenomenon is quite significant. Possible explanations include the accumulation of score/trajectory approximation errors over a long horizon, difficulties in decoder adaptation, or a more fundamental trade-off between making the latent distribution closer to a Gaussian and generative quality. In particular, if the authors measure the actual $p_T$-to-prior discrepancy for each value of $T$, they would be able to distinguish between these possibilities. For example, if the actual latent distribution becomes closer to a Gaussian as $T$ increases but the FID deteriorates, this would suggest a tension between Gaussianization itself and generative quality. Conversely, if the latent discrepancy also worsens, this would serve as evidence that the finite-capacity trajectory/score approximation fails to properly follow the theoretical WGF. 9. The empirical evaluation remains relatively limited. The experiments are limited to three 256×256 unconditional image datasets. The authors themselves acknowledge this as a limitation. Furthermore, the reconstruction FID deteriorates across all datasets, increasing from 0.89 to 2.04 on CelebA-HQ. Therefore, a trade-off between generation and reconstruction, as well as validation in larger-scale settings, is necessary. ### References [1] Dai, B. and Wipf, D. Diagnosing and Enhancing VAE Models. ICLR 2019. [2] Böhm, V. and Seljak, U. Probabilistic Autoencoder. TMLR 2022. [3] Wehenkel, A. and Louppe, G. Diffusion Priors in Variational Autoencoders. ICML Workshop on Invertible Neural Networks, Normalizing Flows, and Explicit Likelihood Models, 2021. [4] Vahdat, A., Kreis, K., and Kautz, J. Score-based Generative Modeling in Latent Space. NeurIPS 2021.
Questions for authors

Q1. Relationship to prior two-stage and latent score/diffusion models. Can you explain more clearly the relationship between this paper and Two-Stage VAE [1], Probabilistic Autoencoder [2], Diffusion Priors in VAEs [3], and, in particular, LSGM [4]? What is the key conceptual and algorithmic novelty compared to these prior studies?

Q2. Controlled latent diffusion/score baseline. Can you provide results from training a conventional latent diffusion/score model using the same pretrained encoder/decoder, latent dimensionality, network capacity, and a similar training budget? In particular, a comparison between the latent diffusion/score model (with the same decoder adaptation) and PC-VAE is important.

Q3. Why freeze the encoder? If the effective encoder after correction is ultimately Eeff=ΨT∘EE_{\mathrm{eff}}=\Psi_T\circ E, why is it necessary to freeze the original encoder EE and train a separate trajectory network? Can you provide a comparison with a simpler baseline that directly fine-tunes the encoder and decoder while aligning the aggregated latent distribution with a Gaussian prior? Such a comparison appears crucial for determining what additional benefits the WGF/Picard framework offers over simple latent representation fine-tuning.

Q4. Component ablations. Can you compare the following: decoder adaptation only; latent correction without decoder adaptation; conventional latent score/diffusion correction combined with decoder adaptation; and PC-WGF combined with decoder adaptation?

Q5. Prior matching of the effective encoder. One-step generation appears to rely critically on the learned finite-TT pTp_T being sufficiently close to qq. Theorem 1 and Theorem 2 do not directly quantify the pTp_T-to-qq mismatch in actual finite-capacity learned models. Is it possible to measure and report the discrepancy between the actual learned effective-encoder distribution and N(0,I)N(0,I) using independent metrics? If so, reporting the discrepancy such as sliced-Wasserstein or MMD along with the generation FID as TT progresses through Picard iterations would allow us to directly verify the mechanism claimed by the method.

Q6. Can you provide a held-out Gaussianity/prior-matching diagnostic that does not depend on the score network? Furthermore, is there a correlation between the reduction in prior discrepancy as Picard iterations progress and the improvement in FID?

Q7. Approximate Picard iteration. Is it possible to obtain a perturbation bound for convergence/stability even when learned score error and trajectory-network approximation error are present? For example, for a finite-time learned distribution, W2(p^T,q)W_2(\hat p_T,q) can this be decomposed into the ideal-WGF finite-time error and the neural/Picard approximation error to establish a bound?

Q8. Terminal time TT. Can you report the actual latent-prior discrepancy at each TT? This should allow us to distinguish whether this phenomenon arises from (i) accumulated approximation error, (ii) decoder adaptation error, or (iii) a trade-off between stronger Gaussianization itself and generative quality.

3 comments
AuthorsOpenReview ↗

Official Response to Reviewer HPza - part 1

Dear Reviewer HPza,

Thank you for highlighting that our method provides a principled and mathematically consistent framework for latent correction. We are glad that you found the proof of convergence of the smoothed KL WGF (circumventing the usual PL inequality), and the semi-implicit Picard iteration analysis as interesting theoretical contributions. We have incorporated your suggestions in the revised draft (new additions are colored green), and we address your questions and concerns below.

Weaknesses

W9. The empirical evaluation remains relatively limited... Furthermore, the reconstruction FID deteriorates across all datasets... Therefore, a trade-off between generation and reconstruction, as well as validation in larger-scale settings, is necessary.

For the 256x256 experiments we have shown the trade-off between generation and reconstruction in Tables 2 and 3 of the main text. We also analyzed this trade-off on a smaller scale experiment on CelebA-HQ with 64x64 resolution, and with latent size 4x4x4 (details of which can be found in Appendix I.7.2), and we observed that PC-VAE actually had both better generation FID and slightly better reconstruction FID than the pre-trained VAE (i.e., there is no trade-off). We summarize the results below.

MethodGeneration FID ↓Reconstruction FID ↓
VAE52.251.54
PC-VAE after 20 iterations + 10k final decoder steps5.671.40

We agree that a larger-scale validation would be useful. However, due to limited compute budget, we are unable to do this over the rebuttal period and leave this as future work.

Questions

Q1 (combined with W1) The novelty relative to two-stage autoencoders and latent score/diffusion models is not sufficiently established... Can you explain more clearly the relationship between this paper and Two-Stage VAE, PAE, Diffusion Priors in VAEs, and LSGM?

Thank you for highlighting these related works. We summarize the key differences between these works and our method below:

  • Two-Stage VAE: Here, a VAE is first trained on the data distribution, and subsequently a second VAE is trained on the latent distribution of the first-stage VAE. Both these VAEs are trained using standard ELBO losses. Generation involves sampling a Gaussian in the latent space of the second-stage VAE, and subsequently applying the second-stage decoder, followed by the first stage decoder. Two-stage VAE addresses the mismatch between the latent distribution and standard Gaussian by fitting a more expressive prior (i.e., a second stage VAE), whereas our algorithm (PC-VAE) addresses this by learning the smoothed KL WGF which transports the original latent distribution toward standard Gaussian.

  • Probabilistic Autoencoder (PAE): PAE first trains an autoencoder on the data distribution, and subsequently uses Normalizing Flow (discussed in Appendix A) to transport the latent distribution to a Gaussian. Thus, PAE fits a generative model in the latent space, similar to Latent Flow Matching. Like our method, PAE addresses the mismatch between the latent distribution and standard Gaussian by learning a transport. The main difference is in how the transport is learnt; our method achieves this by learning the smoothed KL WGF using semi-implicit Picard iteration, while PAE does so using normalizing flow.

  • Diffusion Priors in VAEs: Here, a VAE is trained end-to-end using an ELBO, with the main difference being that the latent prior is taken to be a diffusion prior instead of standard Gaussian. Generation involves sampling a Gaussian zTz_T, and running the reverse diffusion process to get z0z_0, followed by applying the decoder. As with Two-Stage VAE, this work does not match the latent distribution to a Gaussian prior (rather it fits a diffusion prior), whereas PC-VAE does. Additionally, unlike PC-VAE, Diffusion Priors in VAEs is not a framework for one-step generation.

  • LSGM: LSGM broadly follows the same framework as Diffusion Priors in VAEs, in that they model the latent distribution using a score based generative prior, and train the VAE end-to-end using the resulting ELBO. Therefore, the same conceptual differences apply. Additionally, while both PC-VAE and LSGM learn a score network, the way they are used is different: LSGM uses the learnt scores during the generative process to sample from the latent prior through reverse time-dynamics, whereas PC-VAE learns a score network in order to then perform semi-implicit Picard iteration, and is not used during the generative process.

We have now added these references to our revised draft, and cited them in Appendix A. We emphasize that while the problem of addressing the mismatch between the latent distribution and the desired prior is by itself not new, our novelty lies in the mechanism of doing so (learning a smoothed KL WGF using semi-implicit Picard iteration, followed by decoder adaptation), and accompanying theoretical guarantees.

AuthorsOpenReview ↗

Official Response to Reviewer HPza - part 2

Q2 (combined with W1) The current experiments lack a controlled baseline that trains a conventional latent diffusion/score prior on the same pretrained autoencoder... Can you provide results from training a conventional latent diffusion/score model using the same pretrained encoder/decoder, latent dimensionality, network capacity, and a similar training budget? In particular, a comparison between the latent diffusion/score model (with the same decoder adaptation) and PC-VAE is important.

We have now included a comparison with conditional flow-matching with a rectified-flow (RF) path, which is well known to be a reparameterization of the OU flow, performed on the same pretrained auto-encoder and in a compute-matched setting, in Table 11 of Appendix I.7.1. Additionally, we also use the learnt conditional flow model to obtain latent-noise pairs (x,z)(x,z) by numerically simulating the reversal of the flow, following which the decoder is adapted from zz. We test two variants: one where the flow reversal is initialized from the exact latent (t0=1t_0 = 1), and one where the latent xx is first noised to 0.9x+0.1z0.9x + 0.1z where z∼N(0,I)z \sim \mathcal{N}(0,I)), and the flow reversal is initialized from the noised samples, starting from t0=0.9t_0 = 0.9. This comparison is present in Table 12 of Appendix I.7.1. We summarize the results below:

MethodGeneration FID ↓
PC-VAE9.95
Flow Matching (50 NFE)18.25
Flow Matching (100 NFE)17.25
Rectified flow inversion + decoder fine-tuning (t0=1t_0=1)67.51
Rectified flow inversion + decoder fine-tuning (t0=0.9t_0=0.9)12.16

Q3 (a). Why freeze the encoder? If the effective encoder after correction is ultimately Eeff=ΨT∘EE_{eff} = \Psi_T \circ E, why is it necessary to freeze the original encoder EE and train a separate trajectory network?

In our proposed latent correction formulation, we require the original encoder to be frozen in order to supply samples X0X_0 from the initial latent distribution p0p_0 (which are subsequently required by the PC-VAE algorithm, since p0p_0 is the initial condition of the WGF). Directly fine-tuning the original encoder prevents us from obtaining the samples X0X_0. Additionally, our formulation specifically requires training a time-parameterized network in order to learn the trajectories of the smoothed KL WGF, and standard auto-encoder training typically does not have time conditioning for the encoder.

Q3 (b) [combined with W3] It seems necessary to compare this approach with a simpler baseline that applies regularization to align the aggregated latent distribution with a Gaussian distribution while directly fine-tuning the encoder and decoder. Can you provide such a comparison?

We have now provided a comparison with WAE-MMD (a direct latent regularization method) in Table 14 of Appendix I.7.2, under a matched setting (i.e., similar training steps, and using the same pre-trained autoencoder) on CelebA-HQ with 64x64 resolution (with latent size 4x4x4), and we summarize the results below.

MethodGeneration FID ↓Reconstruction FID ↓
VAE52.251.54
WAE-MMD13.811.15
PC-VAE after 5 iterations12.118.73
PC-VAE after 10 iterations6.773.08
PC-VAE after 20 iterations5.471.68
PC-VAE after 20 iterations + 10k final decoder steps5.671.40

Q4 (combined with W2) Can you compare the following: decoder adaptation only; latent correction without decoder adaptation; conventional latent score/diffusion correction combined with decoder adaptation; and PC-WGF combined with decoder adaptation? I believe these controlled comparisons are necessary to isolate their effects.

We report the requested ablations below, on CelebA-HQ with 256x256 resolution:

MethodGeneration FID ↓
Decoder adaptation only139.43
Latent correction without decoder adaptation136.50
Rectified flow inversion + decoder fine-tuning (t0=1t_0=1)67.51
Rectified flow inversion + decoder fine-tuning (t0=0.9t_0=0.9)12.16
PC-VAE9.95

Our understanding is that “latent correction without decoder adaptation” refers to learning the appropriate Gaussianizing transport, but not adapting the decoder, and that the generative pipeline is still sampling a Gaussian and applying the decoder. Please let us know if this is not what you meant.

Ablations (i) and (ii) do not improve the generation FID over the VAE, which is consistent with the pre-trained VAE latent distribution not being close to standard Gaussian.

AuthorsOpenReview ↗

Official Response to Reviewer HPza - part 3

Q5 + Q6 (combined with W5) The existing latent-distribution diagnostics are relevant but insufficient to establish the required prior matching... It would be more convincing to provide distributional metrics independent of the score network... Is it possible to measure and report the discrepancy between the actual learned effective-encoder distribution and N(0,I)\mathcal{N}(0,I) using independent metrics? If so, reporting the discrepancy such as sliced-Wasserstein or MMD along with the generation FID as progresses through Picard iterations would allow us to directly verify the mechanism claimed by the method.

We have now added Figure 9 (Appendix I.7.2), where in addition to the Fisher information (estimated by the score network), we also plot the (normalized) Frobenius norm of the empirical covariance matrix minus identity, sliced Wasserstein and squared MMD with respect to standard Gaussian, and also the generation FID, as a function of number of Picard iterations during training (performed on CelebA-HQ with 64x64 resolution and a 4x4x4 latent space). We observe that there is indeed a correlation between reduction of the various prior discrepancy metrics and the improvement in FID. This strengthens our belief that the improvement in generation comes from better alignment of the latent distribution towards standard Gaussian.

Q7 (combined with W4 + W6) No direct guarantee regarding the prior mismatch in actual finite-capacity neural PC-VAEs is obtained... In actual learned models, there are finite Picard iterations, score estimation error, trajectory approximation error... Therefore, a robustness analysis of how score-estimation error and trajectory-fitting error propagate through Picard iterations would significantly strengthen the theoretical claim... Is it possible to obtain a perturbation bound for convergence/stability even when learned score error and trajectory-network approximation error are present?

Thank you for the interesting question. We had indeed previously obtained a robustness guarantee for our algorithm (i.e., accounting for score and trajectory network errors), however we did not include it earlier due to lack of space in the main paper. We have now included this result in the revised draft (Theorem 3, Appendix H), and we briefly explain the result below: If we consider “imperfect” semi-implicit Picard iterates x^tn\hat{x}^{n}_t, where the residual

ddtx^tn+1−(−a2x^tn+1−a,g(x^tn,px^tn))\frac{d}{dt}\hat{x}^{n+1}_t - (- a^2\hat{x}^{n+1}_t - a\\,g(\hat{x}^{n}_t,p_{\hat{x}^{n}_t}))

has an average squared error (over t∈[0,T]t \in [0,T]) of η2\eta^2 for every iteration, then our new result says that x^tn\hat{x}^n_t remain “close” in L2L^2 to the exact semi-implicit Picard iterates xtnx^n_t; this “closeness” is a function f(ηT)f(\eta T), which goes to 00 as η→0\eta \rightarrow 0.

Q8 (combined with W8) Theoretically, an ideal WGF converges to a Gaussian prior as increases... However, as shown in Table 9, as increases from 5 to 7.5 and 10, the generation FID consistently worsens... Can you report the actual latent-prior discrepancy at each T? This should allow us to distinguish whether this phenomenon (i.e., worsening FID with increasing T) arises from (i) accumulated approximation error, (ii) decoder adaptation error, or (iii) a trade-off between stronger Gaussianization itself and generative quality.

The latent discrepancies across different T are present in Table 10 (Appendix I.5), and are reported below (trained on CelebA-HQ with 256x256 resolution; and Fisher information is estimated using the learnt score network).

MetricN(0,I)\mathcal{N}(0,I) floorT = 5T = 7.5T = 10
Sliced Wasserstein0.0160.0680.0700.071
Squared MMD3e-61.2e-31.17e-31.33e-3
Fisher Information-10.049.8610.28

From these metrics, there appears to be no significant “Gaussianization” after T = 5.0. Since the decoder adaptation pipeline is the same for all T, and stronger Gaussianization is indeed correlated with generative quality (as shown in Figure 9, Appendix I.7), we think that the most plausible underlying reason for worsening of FID with increasing T is (i), i.e., error accumulation in the score and trajectory networks over a long time horizon, and not (ii) or (iii).

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.