- Rating
- 2
- Confidence
- 2
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
The paper shows why the negative ELBO of VAEs trained at increasing information levels first falls and then rises (a "V-shape") while reconstruction keeps improving. By building , they attribute the rising branch to aggregate total correlation (TC). They use 320 encoders trained with KL targets of 4.4–10 nats on a six-factor synthetic task with the decoder fixed to the true generator. All decomposition terms are computed on the empirical aggregate posterior of 1,024 test inputs.
Critical strengths
- The proof is clean, and the paper states the limits of each result openly.
Critical weaknesses
- I am slightly confused by the TC attribution. From my understanding, all terms use the empirical aggregate posterior of test inputs, so nats. The minima and the rising branch in Fig. 1 lie at –, just below this, while the KL target goes up to 10 nats; by the reported slopes, each extra nat of costs about 6.2 nats of KL. Once the empirical MI saturates near , extra rate moves into the marginal KL, which contains [1, 2]. I checked this with the estimator of App. E.2 on a code with independent coordinates (, , so population TC ). With , TC rises 0.15 → 1.36 → 2.40 nats as goes 4.0 → 6.3 → 6.7 (slope ≈ 5 between and 6.9), close to Fig. 1(c) and ; with , the same noise levels give 0.03 → 0.48 → 1.03 nats. App. E.1 notes this effect only for the interventions, and is never varied. The V-shape itself does not depend on , but its attribution to dependence among code coordinates is not supported.
[1] M. D. Hoffman and M. J. Johnson. ELBO surgery: yet another way to carve up the variational evidence lower bound. NIPS 2016 Workshop on Advances in Approximate Bayesian Inference, Barcelona, 2016. [2] A. Bozkurt, B. Esmaeili, J.-B. Tristan, D. H. Brooks, J. G. Dy, and J.-W. van de Meent. Rate-regularization and generalization in variational autoencoders. In Proceedings of the 24th International Conference on Artificial Intelligence and Statistics (AISTATS), PMLR 130:3880–3888, 2021. arXiv:1911.04594.
Questions for authors
Please recompute Fig. 1 and Fig. 4(a,b) with larger evaluation sets (e.g., = 4k, 16k, 60k) or a population-consistent estimator. Does still hold?