acceptodds
Withdrawn from ICLR 2027

A Minimum-Variance Perspective on the Encoder and ELBO in VAEs

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Variational Autoencoders (VAEs) are widely used for generative modeling, denoising, super-resolution, and compression, and are characterized by their encoder-decoder architecture. Yet, from the perspective of the generative model, the encoder is not required: the marginal likelihood is defined solely by the decoder and the latent prior, while the encoder introduces additional trainable parameters. Then, *why should an encoder be part of a generative model?* In this paper, we show that the encoder-decoder structure of VAEs can be derived from a minimum-variance principle and it is not only an inherited architectural choice. Starting from weak assumptions and considering the finite-budget Monte Carlo estimation of marginal likelihoods, we use a classical result from importance-sampling theory to derive the variance-minimizing proposal distribution. This proposal is input-dependent, and its amortized implementation necessarily induces an encoder that maps observations to a distribution over latent variables , followed by a decoder that generates from . Thus, the encoder-decoder structure emerges as a variance-optimal construction for likelihood estimation. We further show that the ELBO is consistent with the same principle: beyond providing a tractable lower bound on the marginal log-likelihood, its optimization is aligned with reducing the variance of the corresponding importance-sampling estimator and with approaching the variance-minimizing proposal. This establishes a common statistical foundation for both the VAE architecture and its training objective. Experiments on controlled latent-variable benchmarks confirm the predicted effect: encoder-based models achieve lower importance-sampling estimator variance and higher likelihood than encoder-free alternatives, with the advantage increasing as the latent dimensionality grows.

Primary area:
probabilistic methods (Bayesian methods, variational inference, sampling, UQ, etc.)
Keywords:
VAE, Importance Sampling, Variance reduction

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_iN1YOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
3
Soundness
1
Read review
Paper summary

The paper asks why a VAE needs an encoder when the generative model is fully specified by pθ(z)p_\theta(z) and pθ(x∣z)p_\theta(x|z). Its answer runs as follows. Estimating pθ(x)p_\theta(x) by sampling z from the prior has high variance. Importance sampling (IS) with an x-dependent proposal qx(z)q_x(z) reduces it, and by the classical result of Kahn & Marshall (1953) the variance-minimizing proposal for a non-negative integrand is the posterior pθ(z∣x)p_\theta(z|x) (Corollary 4.3). An amortized parametric approximation to this proposal is then identified with the VAE encoder (Claim 4.1). Section 5 observes that the ELBO is also maximized at the posterior, and that the IS variance equals to their Equation (13). From this the authors conclude that ELBO training is "aligned" with the minimum-variance principle. The experiments use a synthetic Grid-Squares dataset with discrete latents (I = 2, ..., 5). They compare an encoder-based model with an encoder-free model that samples from a learned prior. The encoder-based model has lower relative importance-weight variance, higher estimated log-likelihood and cleaner samples, and the gap grows with I.

Critical strengths
  1. The question is easy to state, and the paper is short and readable. The derivations in Sections 4 and 5 are correct as stated.

  2. Section 5 is careful to say that the ELBO and the variance criterion are different objectives (reverse KL versus χ2\chi^2) that share an optimum. It does not claim they are equivalent.

  3. Grid-Squares gives direct control over the size of the latent space, and the trend in Figure 3 agrees with what importance-sampling theory predicts.

Critical weaknesses

My main concern is novelty. I could not find a technical or conceptual result in the paper that is not already standard, and the experiments are not strong enough to make up for this.

Questions for authors
  1. Which result in the paper do you consider new, beyond Kahn & Marshall (1953) and the decomposition ELBO = log p(x) − KL? How does your view of the encoder differ from reweighted wake-sleep and inference compilation, which already train an amortized network as an IS proposal for learning the generative model?

  2. What does the minimum-variance principle say about amortized versus non-amortized x-dependent proposals, such as per-datapoint VI or MCMC? If amortization can only add error, in what sense is the encoder derived rather than chosen?

  3. Can you prove a statement that connects ELBO optimization to IS variance inside a restricted variational family, for example a bound on χ2(pθ(z∣x)‖qϕ)\chi^2(p_\theta(z|x) ‖ q_\phi) in terms of the KL gap under stated conditions? Without such a result, what does "aligned" mean beyond sharing an optimum?

  4. In Grid-Squares the encoder-free model's posterior and marginal likelihood can be computed exactly. Could you report (i) exact log pθ(x)p_\theta(x) for both trained models and (ii) an encoder-free model trained by exact maximum likelihood or EM?

  5. What is the training objective of the encoder-free model, and how many samples per data point does it use? Which prior enters the KL term of the encoder-based model, and is it the same distribution used for the samples in Figure 4?

  6. How is "relative variance" computed, and what do the error bars in Figure 3 represent (variation over test images or over seeds)? How many seeds were run?

  7. Do the conclusions hold on a standard benchmark with continuous latents, for example binarized MNIST with a Gaussian latent? A comparison of encoders trained with the ELBO against encoders trained to minimize χ2\chi^2 directly would test the Section 5 claim.

Reviewer_PswfOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

The paper interprets the VAE encoder as an amortized approximation to the variance-minimizing importance-sampling proposal for marginal-likelihood estimation. Applying the classical optimal-proposal result, it identifies the exact posterior as the zero-variance proposal. It then connects this interpretation to the ELBO by observing that ELBO maximization and importance-sampling variance minimization share the posterior as their unrestricted optimum, while acknowledging that their respective reverse-KL and forward-chi-square criteria differ away from that optimum. Experiments on a synthetic Grid-Squares dataset compare an observation-dependent categorical encoder with an unconditional learned-prior baseline, using a common decoder architecture. The encoder-based model achieves lower empirical relative importance-weight variance, higher importance-based log-likelihood estimates, and more coherent qualitative outputs as the number of latent components increases

Critical strengths
  1. A clear and mathematically grounded explanation of the encoder’s computational role. The paper usefully separates the generative model from the auxiliary inference procedure and explains why observation-dependent proposals can improve likelihood estimation. The posterior’s zero-variance property provides an intuitive statistical interpretation. I also appreciate that Section 5 explicitly distinguishes the reverse-KL objective underlying the ELBO from the chi-square criterion governing importance-sampling variance. This makes the exposition potentially useful as an educational synthesis.

  2. A controlled experimental setting with interpretable latent structure. The Grid-Squares construction allows the number of independent latent components to vary while retaining a simple local generative mechanism. The reported results support the narrower claim that a learned conditional proposal can outperform joint prior sampling in this setting. Such controlled experiments are appropriate for investigating the proposed mechanism, although the present comparisons do not fully isolate it.

Critical weaknesses
  1. The distinct contribution relative to closely related work is insufficiently established. The central motivation has direct precedents. Reweighted Wake-Sleep [1 already identifies the posterior as the minimum-variance, zero-variance proposal for marginal-likelihood estimation and explicitly motivates training an inference network to track it for variance reduction. CHIVI [2] derives the relationship between likelihood-estimator variance and forward-chi-square minimization. VIS [3] uses this criterion to train proposals while learning generative models. Furthermore, the already-cited work of Finke and Thiéry [4] develops connections between variance-oriented proposal adaptation and VAE-family gradient estimators. These methods use different training objectives, but they substantially anticipate the motivation and statistical relationships presented here. The submission does not provide a new algorithm, a characterization of optimal proposals within a restricted encoder family, or a computational-budget result that distinguishes it from these antecedents.

  2. The practical minimum-variance interpretation is not established by the theory or isolated by the experiments. The shared-optimum argument holds over unrestricted proposal distributions. In contrast, an ELBO-trained encoder minimizes reverse KL within a restricted, amortized family, and its optimum need not minimize importance-sampling variance. The methods train different generative models and evaluate each using its own proposal, so higher estimated log likelihood can reflect either a better generative model or a smaller proposal-dependent evaluation bias.

[1] Bornschein, J., and Bengio, Y. Reweighted Wake-Sleep. ICLR, 2015. [2] Dieng, A. B., Tran, D., Ranganath, R., Paisley, J., and Blei, D. M. Variational Inference via χ-Upper Bound Minimization. NeurIPS, 2017. [3] Li, C., Wang, Y., Li, W., and Wu, A. Forward χ² Divergence Based Variational Importance Sampling. ICLR, 2024. [4] Finke, A., and Thiéry, A. H. On Importance-Weighted Autoencoders. 2019.

Questions for authors
  1. Clarification of the contribution. What specific theoretical result or conceptual insight extends the minimum-variance inference-network motivation in Reweighted Wake-Sleep, the variance-based proposal objectives in CHIVI and VIS, and the connections developed by Finke and Thiéry?

  2. Clarification and verification of the reported experiments. What are the exact training objectives, training sample counts, and generative priors for the two models? Under the factorization described in Appendix A, can the reported likelihood differences be checked using exact patchwise marginalization, and can proposal variance be compared with the generative model held fixed? If exact marginalization is not applicable to the implemented model, which dependency prevents it? These checks would help determine whether the current results substantiate the claimed mechanism.

Reviewer_iUDnOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

The article proposes a novel view on the role of the encoder in a Variational Autoencoder through the lens of minimum-variance importance sampling to approximate marginal likelihoods. A connection to the well-known evidence lower bound ELBO is given from this perspective. Numerical experiments on controlled latent variable benchmarks show the importance of using encoder-based models to achieve lower importance sampling estimator variance.

Critical strengths

The use of the minimum variance proposal seems to be novel to explain the use of the use of ELBO in well-known methods such as EM and VAE.

The controlled Grid-Squares dataset is also an interesting contribution.

Critical weaknesses

Conceptually, the minimum variance proposal does not seem to give a novel way to solve the problem of max E log p_theta(x), beyond VAE.

Numerical results are not clearly explained, and further improvements might be needed.

Questions for authors

How does the encoder-based method work in Section 6.2? Is it computed using VAE or other approaches?

The result in Fig 3 for computing the variance of your estimator of the marginal log-likelihood is not so clear. It seems that your estimation of the log-likelihood of log p_theta(x) is based on a tight lower bound; what does it mean to be tight if you use M samples of x? It is therefore unclear why the proposed dataset could give better numerical analysis of the estimation variance.

What does the amortized inference in Section 6.2 stand for? Are you using only 1 sample x to learn theta? It is not clear. What is the training set size to learn theta?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.