acceptodds
Under review as a conference paper at ICLR 2027

V-Shaped Negative ELBO in VAEs: Rate Allocation and Aggregate Dependence

Abstract

We uncover a counterintuitive pattern in variational autoencoders: in a controlled benchmark, reconstruction improves as encoder mutual information increases, yet the negative ELBO first decreases and then increases. Decomposing this objective reveals that aggregate total correlation (TC), a cost arising from dependence among code coordinates, grows enough to outweigh the combined improvement in all remaining terms, including reconstruction. Explaining this reversal requires understanding both the reconstruction benefit of additional information and the accompanying dependence cost. Our geometric analysis of optimal Gaussian coding quantifies how reconstruction gains diminish. To characterize the competing cost, we increase information by reducing Gaussian noise around a fixed representation and derive an exact law governing TC growth. Within this noise family, we establish an interior global minimum of the negative ELBO, bound the minimizing noise precision, and give sufficient conditions for uniqueness. Experiments support the predicted allocation of information and TC growth, while complete objective derivatives locate minima along controlled noise interventions. Two general image benchmarks exhibit similar V-shaped responses along measured KL-rate sweeps. These findings explain how increasing dependence can reverse the objective-level benefit of better reconstruction.

Primary area:
interpretability and explainable AI
Keywords:
Variational Autoencoders, Mutual Information, Evidence Lower Bound, Total Correlation, Rate–Distortion Theory.

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_ZWT1OpenReview ↗
Rating
2
Confidence
2
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

The paper shows why the negative ELBO of VAEs trained at increasing information levels first falls and then rises (a "V-shape") while reconstruction keeps improving. By building −ELBO=DNLL+C+TCagg+DWKL-\mathrm{ELBO}=D_{\mathrm{NLL}}+C+\mathrm{TC}_{\mathrm{agg}}+\mathrm{DWKL}, they attribute the rising branch to aggregate total correlation (TC). They use 320 encoders trained with KL targets of 4.4–10 nats on a six-factor synthetic task with the decoder fixed to the true generator. All decomposition terms are computed on the empirical aggregate posterior of 1,024 test inputs.

Critical strengths
  1. The proof is clean, and the paper states the limits of each result openly.
Critical weaknesses
  1. I am slightly confused by the TC attribution. From my understanding, all terms use the empirical aggregate posterior of N=1,024N=1{,}024 test inputs, so C≤log⁡1024≈6.93C\le\log 1024\approx 6.93 nats. The minima and the rising branch in Fig. 1 lie at C≈6.5C\approx 6.5–6.86.8, just below this, while the KL target goes up to 10 nats; by the reported slopes, each extra nat of CC costs about 6.2 nats of KL. Once the empirical MI saturates near log⁡N\log N, extra rate moves into the marginal KL, which contains TCagg\mathrm{TC}_{\mathrm{agg}} [1, 2]. I checked this with the estimator of App. E.2 on a code with independent coordinates (U∼N(0,I6)U\sim\mathcal N(0,I_6), Z=U+t−1/2εZ=U+t^{-1/2}\varepsilon, so population TC =0=0). With N=1,024N=1{,}024, TC rises 0.15 → 1.36 → 2.40 nats as CC goes 4.0 → 6.3 → 6.7 (slope ≈ 5 between C=6.3C=6.3 and 6.9), close to Fig. 1(c) and sTC=5.219s_{\mathrm{TC}}=5.219; with N=8,192N=8{,}192, the same noise levels give 0.03 → 0.48 → 1.03 nats. App. E.1 notes this effect only for the interventions, and NN is never varied. The V-shape itself does not depend on NN, but its attribution to dependence among code coordinates is not supported.

[1] M. D. Hoffman and M. J. Johnson. ELBO surgery: yet another way to carve up the variational evidence lower bound. NIPS 2016 Workshop on Advances in Approximate Bayesian Inference, Barcelona, 2016. [2] A. Bozkurt, B. Esmaeili, J.-B. Tristan, D. H. Brooks, J. G. Dy, and J.-W. van de Meent. Rate-regularization and generalization in variational autoencoders. In Proceedings of the 24th International Conference on Artificial Intelligence and Statistics (AISTATS), PMLR 130:3880–3888, 2021. arXiv:1911.04594.

Questions for authors

Please recompute Fig. 1 and Fig. 4(a,b) with larger evaluation sets (e.g., NN = 4k, 16k, 60k) or a population-consistent estimator. Does sTC>−snonTCs_{\mathrm{TC}}>-s_{\mathrm{nonTC}} still hold?

Reviewer_BSn4OpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper studies V-shaped negative ELBO behavior in variational autoencoders through local Gaussian rate allocation and aggregate posterior dependence. It introduces a geometry-dependent allocation model, derives the growth of aggregate total correlation under Gaussian noise reduction using the I–MMSE relationship, and establishes conditions for interior optima along a fixed-representation noise family. Synthetic experiments examine factor retention, dependence growth, and objective decomposition. Additional image experiments demonstrate V-shaped negative ELBO curves against the total KL rate.

Critical strengths

The paper provides a theoretical account of how increasing encoder mutual information can improve reconstruction while worsening the negative ELBO. Building on rate–distortion theory and the KL decomposition, it analyzes diminishing reconstruction gains and the competing growth of aggregate TC. For a fixed-representation Gaussian noise family, it derives an exact TC-growth law and establishes an interior negative ELBO minimum, together with sufficient conditions for V-shaped behavior.

Critical weaknesses

1. Limited theoretical novelty and scope.

The analysis builds on the MI–TC–DWKL decomposition previously used in β\beta-TCVAE [1], while InfoVAE [2] already examines how controlling mutual information separately from aggregate-posterior matching affects latent-code usage. However, the additional theoretical insight beyond these foundations appears limited. Although the characterization of when TCaggTC_{\mathrm{agg}} increases strictly with mutual information is useful, the underlying derivative identity in Eq. (14) follows directly from applying the classical I–MMSE relationship [3] to joint and coordinatewise information. Similarly, the component-contribution criterion in Section 2.5 follows from the existing decomposition: it identifies TCaggTC_{\mathrm{agg}} dominance once the component changes are measured, but does not independently predict when such dominance will occur.

Their applicability is also limited by the prescribed noise family, which fixes the encoder mean map and varies a common Gaussian noise scale. Although fixed diagonal covariance up to a shared scale is discussed, this setting does not capture the changing representations, input-dependent learned variances, and jointly learned decoders of ordinary VAE training. Moreover, as the authors acknowledge, Proposition 2 guarantees an interior optimum even for independent latent signals. Thus, an interior optimum alone does not establish TC as the cause of the V-shaped response.

[1] Chen et al. Isolating Sources of Disentanglement in Variational Autoencoders. NeurIPS, 2018.

[2] Zhao, Song, and Ermon. InfoVAE: Balancing Learning and Inference in Variational Autoencoders. AAAI, 2019.

[3] Guo, Shamai, and Verdú. Mutual Information and Minimum Mean-Square Error in Gaussian Channels. IEEE Transactions on Information Theory, 2005.

2. Limited evidence for the broader explanatory claims.

Empirical support for the proposed TC-based explanation remains limited to the controlled synthetic setting. Figures 2(a,b) and 3(a) primarily provide numerical consistency checks. Although Figure 2(c) shows that SS predicts factor-retention rankings, its quantitative connection to ELBO reversal remains unclear, and the correlations pool different rates. Moreover, Figure 3(b) locates minima using the measured decoder response and the full objective derivative; substituting the local Gaussian benefit performs substantially worse. This limits the evidence that the allocation theory can predict objective reversal.

Figures 4(a,b) provide stronger evidence for TC attribution within the controlled synthetic setting, supported by additional checks in Appendix E.4. However, this attribution is not evaluated in the image experiments. Figures 4(c,d) report V-shaped negative ELBO curves against the total KL rate RKLR_{\mathrm{KL}}, consistent with established rate–distortion analyses. Without an MI–TC–DWKL decomposition, these results cannot determine whether TC drives the rising branch.

Questions for authors
  1. Could the authors include the budget equation defining τ(Cloc)\tau(C_{\mathrm{loc}}) in Lemma 1 and briefly explain why τ\tau decreases as ClocC_{\mathrm{loc}} increases? Currently, this equation appears only in Appendix A.

  2. Beyond fixed diagonal rescaling, which results extend to independently varying coordinate precisions or input-dependent learned variances? What additional assumptions are required?

  3. Could the authors explain why DWKL changes so little in the evaluated setting and further analyze its contribution to the increase in negative ELBO at higher mutual information levels?

  4. Could the authors investigate whether aggregate TC growth explains the rising branch of the negative ELBO on image benchmarks with jointly learned encoders and decoders? Beyond explaining this behavior, what practical guidance can these findings offer for VAE training or model selection?

Reviewer_X8ZdOpenReview ↗
Rating
3
Confidence
1
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

This paper studies a V-shaped response of the negative evidence lower bound, or negative ELBO, in variational autoencoders. Reconstruction improves as the encoder retains more information, but the full objective can become worse. The paper studies how aggregate total correlation contributes to this reversal. Total correlation measures dependence among latent coordinates across the data distribution.

The analysis combines local Gaussian rate allocation with a fixed representation under decreasing Gaussian noise. It gives an exact dependence-growth law and conditions for an interior objective minimum. Controlled experiments test the proposed model. Two image datasets also show V-shaped objective responses.

Critical strengths

The paper asks a useful question. It examines which objective component can outweigh the advantage of more information. The complete component accounting helps avoid an explanation based only on two selected terms.

The dependence-growth law has a clear interpretation. Dependence grows when joint observation reduces estimation error more than separate coordinate observations. The strictness condition makes this result more informative than a simple monotonicity claim.

The controlled experiments are careful. They compare components on shared observations and use fixed-model noise interventions. The paper reports uncertainty and distinguishes sampled minima from derivative-based estimates. These checks support the internal consistency of the controlled findings.

Critical weaknesses

The main novelty is narrower than the V-shaped framing suggests. The established rate-distortion trade-off already permits an interior optimum of the unweighted objective. Proposition 2 also gives an interior minimum for independent signals, without aggregate dependence. Therefore, the dependence-growth law and its empirical attribution must supply the main new insight. The image experiments show that the negative ELBO first decreases and then increases. However, they do not show whether increased total correlation explains this increase.

The finite reference distribution limits the central empirical claim. The dependence audit uses a fixed mixture of 1,024 encoder means. Appendix E.1 shows that this construction can produce dependence even when the population coordinates are independent. The paper does not test sensitivity to the reference-set size. Thus, the measured dependence can include effects of the evaluation distribution itself. This issue matters for the proposed explanation of VAE behavior.

Questions for authors

Does the total-correlation attribution near the minimum persist when the reference-set size increases? What evidence separates dependence in the population representation from dependence introduced by the finite reference distribution? What conclusion about VAE design follows from the dependence law beyond the established reconstruction-rate trade-off?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.