- Rating
- 2
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper argues that prior-matching constraints are unnecessary for disentanglement and proposes the InfoMax Autoencoder (IMAE), which optimizes a rate-distortion-like objective . Using the I-MMSE relation, the distortion (conditional entropy) and the rate (mutual information) are expressed through denoising MMSE and Fisher information, so both terms become trainable via a pretrained unconditional diffusion denoiser plus a factorized Gaussian encoder whose guidance field supplies the missing information across noise levels, with no explicit prior required. The paper proves that, under separability, factor independence, scale separation and regularity, a factorized Gaussian encoder assigns one latent coordinate per factor and that factors become informative in order of increasing energy as SNR grows (multiscale disentanglement). Experiments on dSprites report a FactorVAE score of 0.73 and MIG of 0.33, better compression and disentanglement than ELBO training at matched distortion, and CelebA examples illustrate SNR-band structure and semantic traversals.
Critical strengths
- The prior-free objective is a novel and reusable contribution. Using the I-MMSE relation and a denoising decoder to write distortion and rate as differentiable functionals of a guidance field connects information theory, diffusion and disentanglement in a non-incremental way. It isolates compression from prior matching and offers an explanation for previously reported gains in diffusion-based representation learning.
- The multiscale prediction is a concrete, testable structural hypothesis. The claim that scale separation plus a factorized Gaussian encoder yields ordered, noise-indexed disentanglement is falsifiable, and the CelebA SNR-band analysis is a suggestive first step toward validating it.
- The distortion derivation is technically clean. Moving beyond restrictive parametric decoders by using I-MMSE to estimate conditional entropy exactly (for fixed ) is a well-argued improvement over the Barber-Agakov bound used by -VAEs.
Critical weaknesses
- The theory may not transfer to the regime the paper targets. The assumptions (independence, local orthogonality, well-separated energies, regularity) are strong and idealizing; the paper motivates them heuristically from natural-image statistics but does not verify them, so it is unclear how much of Theorem 3.1 survives on realistic data. The exact objective is in practice an approximation (learned denoiser, discretized , an encoder-approximated noisy conditional), blurring the boundary between the identity and its implementation.
- The empirical evidence is too thin for the strength of the disentanglement claim. The quantitative result is essentially one synthetic dataset (dSprites), with an absolute MIG around 0.33 and no controlled comparison against the strongest disentanglement baselines; CelebA is qualitative. Together this does not robustly establish that the method improves disentanglement in general.
- The rate comparison is confounded. The InfoMax versus ELBO comparison matches distortion but yields very different rates, and disentanglement is reported at a single beta; sensitivity to beta, the gamma grid and the encoder variance parameterization is not characterized, which is important because the claim is about the compression-disentanglement relationship.
Questions for authors
- How well do the separability, independence, orthogonality and scale-separation assumptions hold quantitatively for dSprites and CelebA? Can you measure the energy gap between factors and test whether the predicted ordering emerges as the theorem requires? This is central to whether the theory explains the experiments.
- Since the optimized objective uses a learned denoiser, discretized and an encoder-approximated noisy conditional, is there any guarantee or empirical bound on the gap between Eq. 21 and the exact ? Does that gap affect the disentanglement conclusions?
- Can you report a controlled comparison against strong disentanglement baselines (-TCVAE, FactorVAE, -VAE variants and a diffusion-based disentanglement method) with matched reconstruction, multiple seeds and error bars, rather than the relative statement that IMAE outperforms all VAE-based methods?
- How sensitive are the rate, the SNR-band assignment and the disentanglement scores to , the -grid and the parameterization of the encoder variance?
In general, I think the most important things is about experiment. For example, authors mentioned that independent factors are not enough to identify generative factors. Then, I want to see on dSprites and CelebA, the reconstructed image comparison when varying the learned latent factors, how accurate the latent factors related to the actual generative factors (in dSprites), between different baseline methods, i.e., my question 2.
2 comments
Answers to Q3-4
- Controlled comparison against strong disentanglement baselines.
- In the appendix we have included a comparison against VAE and diffusion based representation learning methods on CelebA, where we have measured the FID and TAD as a general assessment of generative quality and disentanglement on a naturalistic dataset. In the interest of time we have only trained one instance of the model on CelebA, and because the generative quality differs widely between methods, we have not matched reconstruction.
- Sensitivity of rate, SNR-band assignment and disentanglement scores to hyperparameters.
- Though we wish to provide a detailed numerical estimate of the model’s sensitivity to these hyperparameters, it is not possible to perform such a grid search given our current compute resources. However, in Fig 1 we provide a preliminary sense of the relationship between rate and distortion as a function of , and the optimal schedule depends heavily on the dataset, as shown by Kong et al, (2023). We agree that the diagonal assumptions we make in our encoder is an important parameter to test.
- Could you specify what you mean by the “parameterization of the variance”? If you mean the architectural choices on the encoder for computing the variance, there is likely little effect on the rate or SNR-band assignment since these are determined by information theoretic quantities, which are independent of this choice.
Answers to Q1-2