- Rating
- 1
- Confidence
- 4
- Informativeness
- 1
- Presentation
- 1
- Soundness
- 1
Read review
Paper summary
The paper introduces MoG-Diff, a model that uses a mixture of Gaussians to organize the latent space. Given this latent space, the model employs a latent interface followed by a decoder and subsequently applies a component-conditional diffusion bridge to generate samples. To ensure compatibility with the bridge construction, the authors use the Doob -transform. MoG-Diff is an unconditional model that does not rely on class labels for generation but instead learns them through the learned latent Gaussian mixture. Furthermore, they introduce a Simplex control mechanism for generation. The authors evaluate the semantic organization of the latent space by analyzing clustering performance, alongside reconstruction and generation quality. Experiments are conducted on MNIST, FashionMNIST, and CelebA.
Critical strengths
- The proposed approach is interesting and has potential, particularly if its motivation and theoretical foundations are better established and explained.
- The paper goes beyond merely generating samples by also aiming to learn meaningful semantic representations of the data and enable controllable generation. This multidimensional evaluation is valuable, as it provides a better understanding of how the model captures semantic groups and latent embeddings, and what the generative process relies on, rather than focusing solely on generative performance. It is also encouraging that the model appears to perform well in both clustering and generation on the evaluated datasets.
Critical weaknesses
- Insufficient experimental evaluation and missing ablations: The experimental evaluation is insufficient to substantiate the contributions of the proposed method.
- Clustering evaluation: The clustering experiments are limited to the relatively simple grayscale datasets MNIST and FashionMNIST, without any evaluation on color image datasets. This is particularly relevant because models often struggle to cluster images based on semantic content rather than color. Moreover, color and semantic content can be correlated (e.g., ships and airplanes often appear against blue backgrounds), making semantic clustering more challenging. Therefore, more challenging benchmarks, such as CIFAR-10 or ImageNet, are missing but necessary.
- FID: The authors use conditional feature FID rather than the more commonly adopted FID metric. What is the motivation for this choice, and where does this metric originate? I would advise including the more common FID metric for better comparability across papers.
- Reconstruction fidelity: Reconstruction fidelity is evaluated against only a single baseline, DBAE. Why not compare it to the other baselines as well, that are already used in the paper?
- Component ablations: The paper lacks ablation studies demonstrating the individual contributions of the proposed components. For instance, why is a diffusion bridge preferable to a standard DDPM? How do the different loss terms and their respective weights contribute to the model? How should these weights be selected in practice? Furthermore, what is the specific purpose of the latent interface, and why is it necessary rather than incorporating its functionality directly into the decoder?
- Semantic organization of the latent space: Is the semantic separation induced by the Gaussian mixture components explicitly demonstrated anywhere? While the paper presents component-specific generations and clustering results, it does not appear to directly analyze or visualize the learned latent embeddings. Such an analysis would help substantiate the claim that the mixture components correspond to semantically meaningful regions of the latent space.
- Unclear presentation and insufficient implementation details: The paper is not particularly well written and is often confusing or unclear. The lack of a coherent narrative makes it difficult to follow the proposed methodology and understand the motivation behind individual components.
- Notation and definitions: Abbreviations and variables should be introduced when they first appear. Several variables are used without proper definitions, making the mathematical formulation difficult to follow.
- Structured encoder: What exactly makes the encoder structured, as claimed in line 196?
- Section 3.3: This section is particularly unclear. Several variables, such as , , and , are not properly introduced or explained, with their definitions only appearing in the appendix.
- Responsibility-controlled pairwise paths: The paragraph introducing responsibility-controlled pairwise paths is particularly difficult to follow. What is the purpose of this construction, and how does it relate to the overall methodology? Furthermore, the variables and in Equation 16 are not properly introduced.
- Learned components: The authors should explicitly state which components are learned, how they are learned, and at which stage of the proposed framework.
- Implementation details: The paper lacks essential implementation details. How are the different networks implemented? What architectures are used, how many parameters do they have, how long are they trained, and what computational resources are required?
Minor comments:
- The related work section could also discuss DiffuseVAE [1] and InfoDiffusion [2], as they are works in a similar line as Diff-AE and TreeDiffusion. In particular, DiffuseVAE shares similarities with your proposed work, where instead of a Diffusion Bridge, they employ a conditional DDPM.
- Line 686: There is a broken citation that should be fixed.
- Equation formatting: The paper repeatedly uses the incorrect formatting "Eq. equation XX" (15 occurrences), which should be corrected throughout.
[1] Pandey, Kushagra, et al. "Diffusevae: Efficient, controllable and high-fidelity generation from low-dimensional latents." arXiv preprint arXiv:2201.00308 (2022).
[2] Wang, Yingheng, et al. "Infodiffusion: Representation learning using information maximizing diffusion models." International conference on machine learning. PMLR, 2023.
Questions for authors
- Conditioning mechanism: The authors state that “MoG-Diff organizes the conditioning latent of a diffusion bridge with a Gaussian mixture.” Does this simply mean that the Gaussian mixture is used to determine the starting point of the diffusion bridge, or does any additional conditioning take place during the generative process?
- Predefined number of mixture components: The number of mixture components must be specified in advance, which is a limitation of the proposed approach. How should be selected in practice, particularly when no prior knowledge about the underlying semantic structure or number of clusters is available?
- Figure 3 and the role of the simplex: The purpose of Figure 3 is unclear. Panels (a) and (b) appear to show standard latent-space interpolations, which have already been demonstrated in previous work, such as DiffuseVAE. Regarding the simplex control, what is the benefit of performing interpolations within a triangle rather than simply interpolating between two images?