- Rating
- 1
- Confidence
- 4
- Informativeness
- 1
- Presentation
- 3
- Soundness
- 1
Read review
Paper summary
The paper asks why a VAE needs an encoder when the generative model is fully specified by and . Its answer runs as follows. Estimating by sampling z from the prior has high variance. Importance sampling (IS) with an x-dependent proposal reduces it, and by the classical result of Kahn & Marshall (1953) the variance-minimizing proposal for a non-negative integrand is the posterior (Corollary 4.3). An amortized parametric approximation to this proposal is then identified with the VAE encoder (Claim 4.1). Section 5 observes that the ELBO is also maximized at the posterior, and that the IS variance equals to their Equation (13). From this the authors conclude that ELBO training is "aligned" with the minimum-variance principle. The experiments use a synthetic Grid-Squares dataset with discrete latents (I = 2, ..., 5). They compare an encoder-based model with an encoder-free model that samples from a learned prior. The encoder-based model has lower relative importance-weight variance, higher estimated log-likelihood and cleaner samples, and the gap grows with I.
Critical strengths
-
The question is easy to state, and the paper is short and readable. The derivations in Sections 4 and 5 are correct as stated.
-
Section 5 is careful to say that the ELBO and the variance criterion are different objectives (reverse KL versus ) that share an optimum. It does not claim they are equivalent.
-
Grid-Squares gives direct control over the size of the latent space, and the trend in Figure 3 agrees with what importance-sampling theory predicts.
Critical weaknesses
My main concern is novelty. I could not find a technical or conceptual result in the paper that is not already standard, and the experiments are not strong enough to make up for this.
Questions for authors
-
Which result in the paper do you consider new, beyond Kahn & Marshall (1953) and the decomposition ELBO = log p(x) − KL? How does your view of the encoder differ from reweighted wake-sleep and inference compilation, which already train an amortized network as an IS proposal for learning the generative model?
-
What does the minimum-variance principle say about amortized versus non-amortized x-dependent proposals, such as per-datapoint VI or MCMC? If amortization can only add error, in what sense is the encoder derived rather than chosen?
-
Can you prove a statement that connects ELBO optimization to IS variance inside a restricted variational family, for example a bound on in terms of the KL gap under stated conditions? Without such a result, what does "aligned" mean beyond sharing an optimum?
-
In Grid-Squares the encoder-free model's posterior and marginal likelihood can be computed exactly. Could you report (i) exact log for both trained models and (ii) an encoder-free model trained by exact maximum likelihood or EM?
-
What is the training objective of the encoder-free model, and how many samples per data point does it use? Which prior enters the KL term of the encoder-based model, and is it the same distribution used for the samples in Figure 4?
-
How is "relative variance" computed, and what do the error bars in Figure 3 represent (variation over test images or over seeds)? How many seeds were run?
-
Do the conclusions hold on a standard benchmark with continuous latents, for example binarized MNIST with a Gaussian latent? A comparison of encoders trained with the ELBO against encoders trained to minimize directly would test the Section 5 claim.