acceptodds
Under review as a conference paper at ICLR 2027

Mind the Gaps: Multiscale Disentanglement through InfoMax Auto-Encoders

Abstract

A primary goal for unsupervised learning is to discover disentangled representations of latent factors. Traditionally, such disentanglement is obtained by enforcing constraints on the prior distribution over latents, but this typically entails a loss of reconstruction quality. Here, we show that such constraints are unnecessary. Leveraging an explicit connection between mutual information and MMSE estimators, we develop a novel objective for information maximization which leads to disentangled representations without reliance on prior-matching constraints. Under specific assumptions on the independence, separability, and scale separation of the generative factors, we prove that this objective, paired with a factorized Gaussian encoder, yields multiscale disentanglement. Numerical experiments on both synthetic and natural image datasets support this theoretical prediction. More generally, our approach shows that the combination of classic information-theoretic principles with the expressive power of modern machine learning offers a fruitful path for the development of new models and algorithms.

Primary area:
unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords:
disentanglement, information theory, unsupervised learning, generative models, inference

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_tsBkOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper argues that prior-matching constraints are unnecessary for disentanglement and proposes the InfoMax Autoencoder (IMAE), which optimizes a rate-distortion-like objective h(X∣Z)+βIe(X;Z)h(X|Z)+\beta I_e(X;Z). Using the I-MMSE relation, the distortion (conditional entropy) and the rate (mutual information) are expressed through denoising MMSE and Fisher information, so both terms become trainable via a pretrained unconditional diffusion denoiser plus a factorized Gaussian encoder whose guidance field supplies the missing information across noise levels, with no explicit prior required. The paper proves that, under separability, factor independence, scale separation and regularity, a factorized Gaussian encoder assigns one latent coordinate per factor and that factors become informative in order of increasing energy as SNR grows (multiscale disentanglement). Experiments on dSprites report a FactorVAE score of 0.73 and MIG of 0.33, better compression and disentanglement than ELBO training at matched distortion, and CelebA examples illustrate SNR-band structure and semantic traversals.

Critical strengths
  • The prior-free objective is a novel and reusable contribution. Using the I-MMSE relation and a denoising decoder to write distortion and rate as differentiable functionals of a guidance field connects information theory, diffusion and disentanglement in a non-incremental way. It isolates compression from prior matching and offers an explanation for previously reported gains in diffusion-based representation learning.
  • The multiscale prediction is a concrete, testable structural hypothesis. The claim that scale separation plus a factorized Gaussian encoder yields ordered, noise-indexed disentanglement is falsifiable, and the CelebA SNR-band analysis is a suggestive first step toward validating it.
  • The distortion derivation is technically clean. Moving beyond restrictive parametric decoders by using I-MMSE to estimate conditional entropy exactly (for fixed ZZ) is a well-argued improvement over the Barber-Agakov bound used by betabeta-VAEs.
Critical weaknesses
  • The theory may not transfer to the regime the paper targets. The assumptions (independence, local orthogonality, well-separated energies, regularity) are strong and idealizing; the paper motivates them heuristically from natural-image statistics but does not verify them, so it is unclear how much of Theorem 3.1 survives on realistic data. The exact objective is in practice an approximation (learned denoiser, discretized γ\gamma, an encoder-approximated noisy conditional), blurring the boundary between the identity and its implementation.
  • The empirical evidence is too thin for the strength of the disentanglement claim. The quantitative result is essentially one synthetic dataset (dSprites), with an absolute MIG around 0.33 and no controlled comparison against the strongest disentanglement baselines; CelebA is qualitative. Together this does not robustly establish that the method improves disentanglement in general.
  • The rate comparison is confounded. The InfoMax versus ELBO comparison matches distortion but yields very different rates, and disentanglement is reported at a single beta; sensitivity to beta, the gamma grid and the encoder variance parameterization is not characterized, which is important because the claim is about the compression-disentanglement relationship.
Questions for authors
  • How well do the separability, independence, orthogonality and scale-separation assumptions hold quantitatively for dSprites and CelebA? Can you measure the energy gap between factors and test whether the predicted ordering emerges as the theorem requires? This is central to whether the theory explains the experiments.
  • Since the optimized objective uses a learned denoiser, discretized γ\gamma and an encoder-approximated noisy conditional, is there any guarantee or empirical bound on the gap between Eq. 21 and the exact h(X∣Z)+βIe(X;Z)h(X|Z)+\beta I_e(X;Z)? Does that gap affect the disentanglement conclusions?
  • Can you report a controlled comparison against strong disentanglement baselines (β\beta-TCVAE, FactorVAE, β\beta-VAE variants and a diffusion-based disentanglement method) with matched reconstruction, multiple seeds and error bars, rather than the relative statement that IMAE outperforms all VAE-based methods?
  • How sensitive are the rate, the SNR-band assignment and the disentanglement scores to β\beta, the γ\gamma-grid and the parameterization of the encoder variance?

In general, I think the most important things is about experiment. For example, authors mentioned that independent factors are not enough to identify generative factors. Then, I want to see on dSprites and CelebA, the reconstructed image comparison when varying the learned latent factors, how accurate the latent factors related to the actual generative factors (in dSprites), between different baseline methods, i.e., my question 2.

2 comments
AuthorsOpenReview ↗

Answers to Q1-2

  1. As is invariably the case for any formal proofs of models of this complexity, the assumptions are a mathematical idealizations of natural-image structure, rather than properties that we expect data to satisfy exactly. Their justification comes from several complementary observations.
    • Natural images exhibit approximately power-law spatial spectra, with amplitude decreasing roughly as inverse spatial frequency (Field, 1987), formalized as structure at multiple scales. In parallel, psychophysical experiments show that different spatial scales provide different information for recognition, including coarse scene organization and finer structural details (Schyns and Oliva, 1994; Oliva and Schyns, 1997). These findings suggest that different image factors become recoverable at different noise levels, although they do not establish the theorem’s stronger requirement of non-overlapping factor-energy bands.
    • Independence likewise has a productive history as a modeling principle: maximizing independence among components of natural images yields localized, oriented features (Bell and Sejnowski, 1997), and extensions to independent feature subspaces recover more invariant representations (Hyvärinen and Hoyer, 2000). Though it is unlikely that semantic features are independent, these results nevertheless motivate our goal of seeking partially independent explanatory factors. Our orthogonality and scale-separation assumptions provide the additional geometric structure pre-conditions that makes the proposed mechanism analytically tractable.
    • For dSprites, independence of the ground-truth factor labels holds by construction under uniform sampling of the complete dataset: every combination of shape, scale, orientation, and position is represented. This also permits controlled comparisons in which one factor changes while the others remain fixed. However, independent sampling does not imply that the corresponding changes in pixel space are orthogonal, nor that their energies occupy distinct bands. This is also difficult to quantify because both aspects require knowledge of the feature map's Jacobian JkJ_{k}, and though we know the ground truth factors, we do not have direct access to the generative model that maps them to the image domain.
  2. Guarantees or bounds on gap between implemented and ideal objective due to implementation details, and their effect on disentanglement.
    • It's true that each of these factors can induce a gap between the implemented objective and the ideal objective, h(X∣Z)+βIe(X;Z).h(X|Z)+\beta I_{e}(X;Z). Both terms are affected by an approximation error in the noisy encoder, but if the encoder approximation is exact (i.e. when pe,γ(z∣xγ)=∫pe(z∣x)p(x∣xγ)dxp_{e, \gamma}(z|x_{\gamma})= \int p_{e}(z|x)p(x|x_{\gamma}) dx, line 279), we recover the true rate term Ie(X;Z)I_{e}(X;Z). Additionally, if the learned denoiser is also optimal and we have no discretization error, we also recover the exact conditional entropy and thus the ideal objective.
    • Bounds: we are able to quantify the numerical error from γ\gamma discretization for information-theoretic diffusion by referencing Kong et al., (2023), who showed that the discrete and continuous density estimators differ only by constants. However, because replacing the exact guidance score induces both a squared-error term and a cross-term, the rate estimate does not offer a one sided bound in the same way.
    • Effect of errors on disentanglement: the effect of the approximation gap affects the representation, and thus how disentangled it is, but not always for the worse. For example, the ideal objective is indifferent between representations that possess equal amounts of information, but our encoder-approximated noisy conditional do not approximate these representations equally well. Instead, because the implemented encoder favors those representations with diagonal covariance structure, the approximation error that is induced by the approximation nevertheless favors disentanglement.
AuthorsOpenReview ↗

Answers to Q3-4

  1. Controlled comparison against strong disentanglement baselines.
    • In the appendix we have included a comparison against VAE and diffusion based representation learning methods on CelebA, where we have measured the FID and TAD as a general assessment of generative quality and disentanglement on a naturalistic dataset. In the interest of time we have only trained one instance of the model on CelebA, and because the generative quality differs widely between methods, we have not matched reconstruction.
  2. Sensitivity of rate, SNR-band assignment and disentanglement scores to hyperparameters.
    • Though we wish to provide a detailed numerical estimate of the model’s sensitivity to these hyperparameters, it is not possible to perform such a grid search given our current compute resources. However, in Fig 1 we provide a preliminary sense of the relationship between rate and distortion as a function of β\beta, and the optimal γ\gamma schedule depends heavily on the dataset, as shown by Kong et al, (2023). We agree that the diagonal assumptions we make in our encoder is an important parameter to test.
    • Could you specify what you mean by the “parameterization of the variance”? If you mean the architectural choices on the encoder for computing the variance, there is likely little effect on the rate or SNR-band assignment since these are determined by information theoretic quantities, which are independent of this choice.
Reviewer_hwHaOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

The paper proposes the InfoMax Autoencoder (IMAE), a representation learning framework that seeks to achieve multiscale disentanglement without explicitly matching the aggregated latent posterior to a prescribed prior distribution. Using the I-MMSE relation and de Bruijn's identity, the authors reformulate the rate-distortion objective into a denoising objective across multiple signal-to-noise ratios (SNRs), combining a diagonal Gaussian encoder with a diffusion-based U-Net denoiser. They provide a theoretical analysis (Theorem 3.1) arguing that a factorized Gaussian encoder paired with scale-separated generative factors yields axis-aligned representations across noise levels, and evaluate the method on dSprites and CelebA.

Critical strengths
  • Conceptual Direction: Exploring the interplay between information-theoretic rate-distortion, multi-scale denoising across SNR levels, and latent disentanglement is an interesting conceptual perspective.

  • overall: I think the paper has very interesting ideas and a novel framing for disentanglement that can be very beneficial to the disentanglement community if presented correctly.

Critical weaknesses

1. Clarity, Presentation, and Notation IssuesHigh-Level Narrative & Architecture Obscurity:

  • The paper requires substantial rewriting to make its core story and methodology easy to follow. From the abstract and introduction, it is difficult to discern what the concrete positionning of the paper is. Furthermore, the narrative around constraints is contradictory: the authors claim to eliminate prior constraints, yet reintroduce strong constraints via InfoMax rate regularization (Eq. 21) and an explicit diagonal Gaussian distribution assumption on the encoder (Eq. 22).
  • Unpolished Notation & Late Definitions: Several mathematical quantities are introduced without clear definitions or context in the * main text:In Eq. (6), I~(X;Z)\tilde{I}(X;Z) is defined inline without prior introduction. In Eq. (13), parameters θ\theta and ϕ\phi appear in L(θ,ϕ)\mathcal{L}(\theta, \phi) before specifying what θ\theta parameterizes in this specific architecture, and x^∗(xγ,γ)\hat{x}^*(x_\gamma, \gamma) is introduced abruptly in Eq. (15). XγX_\gamma and the SNR/noise parameter γ\gamma appear throughout Sections 2.5 and 3, but the main text does not clearly explain how γ\gamma is obtained, scheduled, or discretized when approximating the integral over [0,∞)[0, \infty) in Eq. (21). In Eqs. (16)–(21), it is entirely unclear how the guidance field aϕ(xγ,z,γ)=∇xγlog⁡pe(z∣xγ,γ)a_\phi(x_\gamma, z, \gamma) = \nabla_{x_\gamma} \log p_e(z\vert{}x_\gamma, \gamma) is analytically computed until Eq. (22) belatedly reveals that pe(z∣xγ,γ)p_e(z\vert{}x_\gamma, \gamma) is parameterized as a diagonal Gaussian. In Section 2.3, the statement "For a Gaussian decoder with fixed isotropic covariance, DrecD_{\text{rec}} is proportional to the expected squared error of the decoder mean, up to an additive constant" feels disconnected from the subsequent methodology, as the takeaway is never developed further.

2. Theoretical and Methodological ConcernsRedundancy Between Reconstruction and InfoMax Objectives:

  • A standard autoencoder reconstruction loss already pushes the latent representation to be information-theoretically optimal with respect to MSE reconstruction (e.g., recovering PCA subspaces in the linear case). It is unclear what InfoMax brings to the latent space beyond what a reconstruction objective already enforces—especially since the authors subsequently use the I-MMSE relation to rewrite the mutual information / conditional entropy objective directly back into an MMSE denoising reconstruction loss (Eqs. 11–14, 21). Conversely, balancing distortion with mutual information in Eqs. (4) and (13) is not sufficiently justified, as maximizing mutual information blindly could encourage encoding image features that are unnecessary for reconstruction.
  • Lack of a Valid Bound on the Intractable Objective: The paper discards the standard NELBO in favor of the surrogate loss in Eq. (21), which relies on learned denoisers, score approximations, and discretized integrals over γ\gamma. The authors never prove that their tractable training loss is a valid upper/lower bound on the true intractable objective, which is concerning from an optimization standpoint.
  • Prior Constraints & Theorem 3.1 Assumptions: In β\beta-VAE, prior matching enforces disentanglement because the target prior has mutually independent axes. By relying on an analytic diagonal Gaussian encoder constraint in Eq. (22), IMAE still falls within the paradigm of analytic distributional constraints (similar in spirit to VAE sampling) and remains vulnerable to the failure modes documented in Fruytier et al. (Sculpting the Latent Space with MMD), where the assumed distributional properties are not actually satisfied by the aggregated latent space. Moreover, Theorem 3.1 assumes a priori that the encoder is already factorized while the data generative factors are orthogonal and independent with separated scales, making the theoretical disentanglement guarantee feel circular.

3. Insufficient Baselines, Missing Metrics, and Reporting InconsistenciesWeak Baseline Comparisons: In the main text (Table 1), IMAE is only compared against a single baseline labeled "ELBO" (which is not a standard baseline, but rather the authors' own diffusion-decoder architecture trained with a KL regularizer). The authors should compare directly under controlled conditions against established methods—including β\beta-VAE, β\beta-TCVAE, and approaches that enforce mutually independent latent spaces via alternative constraints, such as Yeats et al. (Disentangling representations through adversarial covariance minimization) and Fruytier et al. (Sculpting the Latent Space with MMD).

  • Limited Disentanglement Metrics: Evaluation on dSprites relies solely on MIG and FactorVAE scores. To thoroughly gauge disentanglement and mutual independence, the authors should report DCI scores as well as unsupervised mutual independence metrics such as the Latent Predictability Score (LPS) across multiple random seeds. Contradictory Numbers & Benchmark Discrepancies:In Section 4.1, the text states that IMAE achieves "a FactorVAE score of 0.73±0.0060.73 \pm 0.006 and a MIG score of 0.33±0.00070.33 \pm 0.0007", which contradicts the numbers reported immediately below in Table 1 (0.653±0.0070.653 \pm 0.007 and 0.210±0.0010.210 \pm 0.001). Additionally, Table 1 labels the metric column "Factor VAE" (a model name) rather than "FactorVAE Score". The reported MIG scores for ELBO/VAE baselines (0.0450.045 in Table 1; 0.050.05 in Table 2) do not align with independently established benchmarks such as Locatello et al. (2019, Table 13), which report dSprites MIG scores between 0.10.1 and 0.20.2 (and reaching ∼0.4\sim 0.4).
Questions for authors
  • InfoMax vs. Reconstruction Optimality: Since minimizing reconstruction error under a bottleneck already encourages learning an information-theoretically optimal representation for reconstructing XX (and your I-MMSE formulation ultimately reduces the objective back to a denoising MSE in Eq. 21), what specific property does the InfoMax framing provide that standard reconstruction + bottleneck regularization does not?

  • Optimization Guarantees & Relation to NELBO: Does Eq. (21) constitute a formal bound on the exact rate-distortion Lagrangian in Eq. (13) or the data log-likelihood? When throwing out the ELBO prior-matching term and approximating the γ\gamma-integral over [0,∞)[0, \infty) via finite discretized sampling, what prevents degenerate scaling or unbounded drift in the aggregated posterior?

  • Training Pipeline & γ\gamma Parameterization: Are the probabilistic encoder pe(z∣xγ,γ)p_e(z\vert{}x_\gamma, \gamma) and the unconditional denoiser x^θ(x,γ)\hat{x}_\theta ( x , \gamma ) trained jointly from scratch, or is the diffusion model pre-trained and frozen? How is the SNR parameter γ\gamma sampled/scheduled in practice, and is it treated as a tuned hyperparameter?

  • Pointwise Conditional Independence vs. Marginal Independence: Regarding Eq. (22), you state that restricting the encoder to a diagonal Gaussian "makes latent coordinates conditionally independent given the input, without requiring the aggregate latent distribution to match a prescribed prior". If conditional independence holds pointwise for all xx, does this guarantee marginal independence of the aggregated posterior pe(z)p_e(z), or can the aggregated latent coordinates remain heavily correlated? Have you verified empirically whether the learned marginal distribution actually satisfies mutual independence (e.g., via LPS or covariance checks, as scrutinized in Sculpting the Latent Space with MMD)?

  • Discrepancies in Table 1 and Baseline Scores: Why do the FactorVAE and MIG scores in the text of Section 4.1 (0.730.73 and 0.330.33) differ from those in Table 1 (0.6530.653 and 0.2100.210), and why is the ELBO MIG score (0.0450.045) substantially lower than standard literature values (e.g., Locatello et al., 2019, Table 13)? Will the authors expand the controlled evaluation to include β\beta-VAE, β\beta-TCVAE, WAE-MMD (Sculpting the Latent Space with MMD), and adversarial covariance minimization (Yeats et al.), evaluated with DCI and LPS across multiple random seeds?

3 comments
AuthorsOpenReview ↗

Answers to reviewer's questions

Thank you for your detailed questions.

  1. InfoMax vs. Reconstruction Optimality:
    • Though denoising and reconstruction are closely related objectives, our version is an integral over all values of γ\gamma, while VAEs use conditional Gaussian decoders with a single fixed scalar variance. This means that the I-MMSE relation is an infinite set of reconstruction losses, each selecting for a particular scale in the image space (in practice, this is numerically approximated with a large but discrete set of γ\gamma values). As such, the model learns its representation based on a better estimate of the underlying data distribution (unconditional diffusion model), and it uses knowledge of γ\gamma to organize the representation in a scale-specific manner.
  2. Optimization Guarantees & Relation to NELBO:
    • Since we implement the distortion (first term in Eq. 21) as a series of mean squared error terms, any approximation error is also non-negative. This translates into an upper bound on the idealized conditional entropy, which is exactly the Barber-Agakov bound (Eq. 6). The rate term is equivalent to the squared norm of the guidance field. If the learned encoder-derived guidance field is equal to the true score of the noisy conditional, we recover the exact rate. Though the error is not necessarily non-negative, its magnitude is bounded from above through Cauchy-Schwarz by the magnitude of the score error and its alignment with optimal guidance.
    • The loss by itself does not prevent scaling or unbounded drift in the aggregated posterior, but using the same encoder network to encode both clean and noisy images prevents mean drift because any global translation biases are inactive during training (i.e. the derivative of the loss with respect to a mean bias term is zero). Moreover, though there is no mechanism to prevent scaling, there is also no incentive for the network to increase or decrease the scale of the mean and covariance.
  3. Training Pipeline & Parameterization:
    • We initialize training using a pre-trained diffusion model, but then we train it jointly with the encoder. We’ve also successfully trained models using a pre-trained frozen diffusion model.
    • We use the same schedule for γ\gamma that is used by the underlying DDPM denoiser, i.e. a cosine noise schedule. We do not tune this hyperparameter, but their optimal values (for estimating the data entropy) is dependent on the dataset and is an empirical question.
  4. Pointwise Conditional Independence vs. Marginal Independence:
    • While we do not have a guaranteed marginal independence of the aggregated posterior pe(z)p_e(z) from pointwise conditional independence for all xx, we have verified empirically whether the each latent exclusively captures a pseudo-label (a proxy for mutual independence) using the TAD score from the NashAE paper (Yeats et al, 2022), on a model trained on CelebA. We get a value of 0.583, compared to 0.042 (AE), 0.000 (VAE), 0.088 (β\beta-VAE), 0.155, (DiffAE), 0.299 (InfoDiffusion), 0.305 (DisDiff), 0.362 (DBAE + TC). We have listed these additional results in the Appendix.
    • The recommended metrics (covariance checks + LPS) would let us check whether the learned marginal satisfies mutual independence, but these methods stem from work that was only recently published (Fruytier et al., Oct 2026). Given the short rebuttal timeline, we are unable to provide these metrics but they can be included in a later revision.
  5. Discrepancies in Table 1 and Baseline Scores:
    • “Why do the FactorVAE and MIG scores in the text of Section 4.1 differ from those in Table 1?”
      • Thank you for catching these discrepancies. The values in the appendix are from a previous set of experiments while the values reported in Table 1 are from our latest model runs. We will update the Appendix to the latest values.
    • “Why is the ELBO MIG score substantially lower than standard literature values?”
      • To match reconstruction performance we chose a value of β\beta that is small, which puts little pressure on the representation to be disentangled. This allows us to see that for a fixed distortion level, the method trained on the InfoMax objective can use a much lower rate than the ELBO, and we see that even at these small values of β\beta, the additional aggregate-posterior-matching term in the ELBO causes the same architecture to exhibit a lower DCI value.
    • “Will the authors expand the controlled evaluation to include β\beta-VAE, β\beta-TCVAE, WAE-MMD, and adversarial covariance minimization (Yeats et al.), evaluated with DCI and LPS across multiple random seeds?”
      • While we aim to expand our comparison to DCI, we note that due to limited time we are unable to cover WAE-MMD and LPS since they stem from a paper that has only recently been published (Oct 2026). We note, however, that DCI has been found to be highly correlated with MIG (see Locatello et al., 2019), which we report in Table 2.

I thank the authors for their responses to my comments which have cleared up some of my questions. Specifically, the γ\gamma schedule, training pipeline, and alingment metrics (TAD score). Due to the late stages of the rebuttal, I do not expect the authors to make any additional experiments at this stage.

However, I still have skepticisms relative to evaluations (specifically [Q5] where β\beta small was chosen in evaluations), infoMax objective, and overall clarity of the paper. As a result, I will maintain my score as things stand.

The authors have explained that the short rebuttal did not give them enough time to address all of my above concerns adequatly. However, if the authors can update the current manuscript's text or concretely explain how they will update (i) the paper's positioning, (ii) the methodology (to be clearer and more readable), and (iii) the limitations of the current evaluations, I would be open to reconsidering my score before the discussion period closes.

Thank you for your thoughtful response. We have updated the manuscript to improve clarity, following your suggestions in W1. Moreover, we have expanded Section 4.1 (Q5) and evaluated our dSprites model on the DCI metric, as requested. We have listed these changes in a general response to all reviewers.

We also wish to respond further to W1 regarding the positioning of the paper.

  • We do not claim that we would eliminate all prior constraints, but rather that we aim to eliminate the prior (in the Bayesian sense) used by the ELBO. Specifically, we state on lines 81-83 that "we develop a constrained information maximization (InfoMax) objective that removes the prior-matching penalty and instead exploits the interaction between encoder factorization and the multi-scale structure of the data." (lines 81-83). This means we still wish to place certain constraints on the model, just not in the form of a prior over the latent variables.
  • It is also worth noting that constraints are necessary in InfoMax (or more generally in autoencoders) to learn meaningful representations; without them the model is incentivized to learn the trivial solution that copies all information in XX into ZZ, i.e. an identity map. With this in mind, our information-theoretic rate constraint differs considerably from the ELBO prior matching penalty both conceptually and in its impact on the learned representation. Our rate penalty places no assumptions on the marginal distribution over the latents (unlike the ELBO, which uses an explicit prior m(z)m(z)), and since the rate is a purely information-theoretic constraint, it allows the model to learn more flexible and disentangled representations (as demonstrated by Fig 1 and Fig 2).
  • Additionally, though we constrain the encoder to be diagonal Gaussian, this architectural inductive bias also does not constitute a prior over the latent variables zz. It is the minimal assumption we make on the mapping between XX and ZZ that allows multi-scale disentanglement to emerge (Thm 3.1). As a result, we do not believe that our narrative around constraints is contradictory but consistent with existing InfoMax (Linsker 1988) and VAE literature.
Reviewer_KVWnOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes an information-theoretic objective function for achieving disentanglement. Traditional VAEs have two terms : a reconstruction term and a KL divergence term. And the KL divergence term can be separated into a mutual information term and a prior matching term. This paper removes the prior matching term of the traditional VAE. However, the mutual information term alone is hard to calculate. So they use the I-MMSE relationship to optimize it. They give the theoretical guarantee of multiscale disentanglement and do some experiments to show the performance of disentanglement.

Critical strengths

Changing the traditional prior KL penalty to a mutual-information regularizer and not needing to match the aggregated posterior to the predefined prior is interesting. It helps to investigate which part of the traditional VAE is necessary for disentanglement. And they provide the theoretical results for disentanglement without matching the prior.

Critical weaknesses

The idea of this paper is to use beta to control the relative cost of retaining information.(line 148). And we can have a tradeoff for reconstruction and compression. But for equation 13, L=h(X|Z)+beta * I(X;Z)=h(X)-I(X;Z) +beta I(X;Z)=h(X)+(beta-1) I(X;Z) . And here h(X) is constant, thus we are maximizing the information or minimizing the information according to the value of beta. There is no tradeoff here. This makes me confused. Authors should elaborate on it more.

Questions for authors

What does equation 13 mean? Is your implementation aligning with equation 13? Do you really have a reconstruction and compression tradeoff in your implementation?

2 comments

We thank the reviewer for their questions.

  • The reviewer is correct that the algebraic expression in Eq. 13 does not in itself define a trade off, since it splits the rate-distortion curve into three disjoint regimes. However, our implementation of the terms does, in two important ways:
    • First, our objective minimizes the Barber-Agakov upper bound and approximates the true noisy conditional with a Gaussian encoder (line 279). These two factors turn the rate-distortion plane into a continuous curve where β\beta corresponds to a meaningful trade-off between compression and reconstruction (see Fig. 1)
    • Second, the two terms in our objective (Eq. 13) place different constraints on the encoder. The distortion term h~(X∣Z)\tilde{h}(X|Z) encourages the encoder to produce a useful guidance field for the denoiser by learning an appropriate metric in the representation space, while the mutual information term Ie(X;Z)I_e(X;Z) minimizes the information accessible to the decoder by reducing the squared norm of this guidance field (Eq. 21). Since both of these properties are continuous values, we get a gradual trade-off between compression and reconstruction as we change β\beta.
  • We have updated the notation of the distortion term in Eq. 13 from h(X∣Z)h(X|Z) to h~(X∣Z)\tilde{h}(X|Z) to make it clear that we are optimizing an upper bound via I-MMSE.

Thanks for the detailed responses. The revision clarifies and fixes the inconsistent score. But I think the experiments still do not directly test the proposed multiscale mechanism. I will keep my score.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Additional experiments and revisions to the manuscript

We performed almost all experiments requested by the reviewers and updated our manuscript based on the reviewers' suggestions. Here is a list of our new numerical results and changes we made to the text:

  • Numerics:
    • We performed the additional experiments requested by reviewers hwHa and tsBk to quantitatively measure disentanglement and image generation quality. We did this for both synthetic (dSprites, results in Appx. A8, Table 2) and naturalistic (CelebA, results in Appx. A10, Table 5) datasets.
    • For dSprites, we measured DCI-D and found that our model is competitive with existing models in the literature (Appx A8). We also bolstered our analysis in Section 4.1 by training an additional InfoMax model to match the rate of the ELBO model (where previously we only matched its distortion). Our results indicate that training the model on InfoMax leads to a more disentangled representation regardless of whether we match the ELBO model's rate or its distortion.
    • For CelebA, we measured FID as a measure of generative image quality and TAD as a measure of disentanglement on binary labels. This provides empirical evidence that our model outperforms the standard VAE-based models and remains competitive with diffusion-based representation learning models, despite using a simpler parameterization for the guidance mechanism.
  • We edited the manuscript for clarity. To make it easier to see where we updated the manuscript, we marked the newly edited sections in red.
    • Following feedback from reviewer KVWn, we updated Eq. 13 to use h~(X∣Z)\tilde{h}(X|Z) rather than h(X∣Z)h(X|Z).
    • Following feedback from reviewer hwHa, we changed the following:
      • We reworded the paragraph in Section 2.3 (lines 149-155) to better explain the formal equivalence between the log-likelihood reconstruction loss in VAEs with the distortion DrecD_{\text{rec}} in rate-distortion.
      • We updated the text to ensure that mathematical quantities and network parameters are properly defined before they are used in an equation. This includes defining: I~(X;Z)\tilde{I}(X;Z) as the Barber-Agakov (variational) bound on mutual information (line 180); x^∗(xγ,γ)\hat{x}^{*}(x_{\gamma}, \gamma) as the optimal unconditional denoiser (lines 210, 222, 295), and x^Z∗(xγ,γ,z)\hat{x}^{*}_{Z}(x_{\gamma}, \gamma, z) as the optimal conditional denoiser (lines 215, 222, 296); parameters θ\theta and ϕ\phi before they appear in L(θ,ϕ)\mathcal{L}(\theta, \phi) (lines 264, 269). - We provided more detail on how γ\gamma is obtained, scheduled, and discretized during training (lines 267-268, lines 343-345).
      • We included a paragraph discussing bounds on estimation errors for the distortion and rate, and conditions when we get equality (lines 348-354). We also included a proof that the magnitude of the error in rate estimation is bounded (Appx. A7).
      • We edited the wording in Section 4.1 to accommodate our new analysis and make it easier for the reader to understand the goals of this analysis.
      • We edited Section 3 for clarity (additional paragraph on lines 259-262, edited lines 294-298).

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.